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Abstract 


This  report  provides  findings  from  the  research  conducted  under  RT-18:  Valuing 
Flexibility  project  during  the  first  seven  month  period.  The  primary  goal  of  this  research 
project  is  to  identify,  develop  and  validate  sound  quantitative  methods,  processes,  and 
tools  that  will  enable  DoD  leadership  and  program  managers  to  make  a  convincing  case 
for  investments  in  system  flexibility  when  acquisition  decisions  are  made. 

The  research  during  this  period  focused  on  identifying  current  quantitative  MPTs  for 
valuing  flexibility  in  DoD  context,  and  delivering  an  initial  capability  to  value 
investments  that  provide  the  flexibility  to  handle  foreseeable  sources  of  change,  using 
easy-to-understand  monetizable  terms,  such  as  the  effect  of  the  investments  on  total 
cost  of  ownership  or  return  on  investment.  We  conducted  a  critical  evaluation  of  the 
theoretical  foundations  underlying  current  approaches,  the  dimensions  of  flexibility, 
measures  of  flexibility,  value  functions,  and  methods  for  incorporating  flexibility  -  both 
at  the  design  phase  and  the  operational  phase,  to  identify  strengths  and  weaknesses  of 
each  approach.  During  this  period,  we  also  explored  the  development  of  an  analytical 
framework  based  on  sound  mathematical  constructs. 

A  review  of  the  current  state-of-the-art  showed  that  there  is  little  unifying  theory  or 
guidance  on  best  approaches  to  measure  flexibility,  quantify  value  of  flexibility  in  a 
prospective  systems  acquisition  or  which  MPTs  work  best  in  which  situations.  In  fact, 
the  analytical  basis  for  defining  and  valuing  flexibility  is  missing.  Considering  this  major 
gap  in  the  current  state-of-the-art  the  primary  focus  of  our  research  activities  is  in 
developing  a  coherent  value  based  definition  of  flexibility  that  is  based  on  an  analytical 
framework  that  is  mathematically  consistent,  domain  independent  and  applicable  under 
varying  information  levels.  This  report  presents  our  advances  in  defining  and 
formalizing  the  value  of  flexibility  and  the  underlying  capability-need-value  architecture 
that  could  form  the  basis  for  developing  tools  that  can  be  used  by  systems  acquisition 
decision-makers  to  conduct  tradeoff  analysis  between  flexibility  and  other  system 
performance  measures  of  interest. 
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1  Project  Overview  and  Goals 


The  primary  goal  of  this  research  project  is  to  identify,  develop  and  validate  sound 
quantitative  methods,  processes,  and  tools  that  will  enable  DoD  leadership  and  program 
managers  to  make  a  convincing  case  for  investments  in  system  flexibility  when 
acquisition  decisions  are  made.  This  research  will  enable  systems  engineers  to 
determine  “the  value  of  flexibility”  in  various  situations,  and  thus  enable  DoD 
acquisition  personnel  to  make  more  informed  decisions  on  “how  much  to  invest  in 
flexibility”.  A  further  goal  is  to  identify  gaps  between  current  capabilities  and  DoD 
needs  for  both  valuing  and  improving  the  flexibility  of  military  systems,  and  to  create  a 
roadmap  for  researching  and  developing  such  capabilities. 

This  report  provides  details  of  the  tasks  completed  under  this  project  during  the  first 
phase. 


1.1  Motivation  and  Need 

Flexibility  is  almost  universally  perceived  as  a  good  thing.  Systems  acquisition  in  the 
DoD  is  no  exception,  where  programs  typically  strive  to  infuse  some  degree  of  flexibility 
into  the  system  being  developed.  It  is  becoming  increasingly  clear  that  future  DoD 
systems  need  to  be  highly  adaptive  to  rapid  changes  in  adversary  threats,  emerging 
technology,  and  mission  priorities,  both  during  development  and  during  operations. 

Traditionally,  however,  complex  DoD  systems  have  been  designed  to  deliver  optimal 
performance  within  a  narrow  set  of  initial  requirements  and  operating  conditions  at  the 
time  of  design.  This  usually  results  in  the  delivery  of  point-solution  systems  that  fail  to 
meet  emergent  requirements  throughout  their  lifecycles,  that  cannot  easily  adapt  to  new 
threats,  that  too  rapidly  become  technologically  obsolete,  or  that  cannot  provide  quick 
responses  to  changes  in  mission  and  operating  conditions.  It  is  possible  to  design 
engineering  systems  with  degrees  of  freedom  such  that  they  exhibit  flexibility  and/or 
robustness  in  future  operating  environments.  However  a  critical  challenge  is  to  assess 
how  much  such  an  upfront  investment  in  design  with  various  degrees  of  freedom 
(margins,  generics,  service-oriented  interfaces,  product-line  architectures)  is  worth  to 
decision-makers,  and  how  this  added  flexibility  impacts  the  various  performance 
attributes  of  the  system. 

To  answer  these  types  of  questions,  we  must  develop  valid  methods  to  measure  the 
value  of  flexibility.  First,  we  need  to  know  how  much  flexibility  we  have  in  a  given 
system,  so  that  we  can  compare  the  design  flexibility  across  systems,  as  well  as  the 
flexibility  of  different  design  options  within  a  given  system.  Despite  its  wide  usage  and 
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high  regard,  there  is  little  consensus  on  even  a  formal  definition  of  flexibility.  Making 
this  task  more  challenging  is  the  fact  that  there  are  a  litany  of  “-ilities”  that  appear  to  be 
closely  related  to  the  notion  of  flexibility,  including  adaptability ,  robustness,  agility, 
changeability,  modularity,  interoperability,  modifiability,  scalability,  and  versatility. 

With  a  better  sense  of  what  is  meant  by  flexibility,  we  can  then  explore  the  notion  of  its 
value.  There  is  a  plethora  of  questions  to  consider.  Is  flexibility  always  a  good  thing  or 
are  there  circumstances  where  its  value  is  diminished?  And  how  precisely  can  we 
measure  that  value?  What  is  known  regarding  the  correlation— either  positive  or 
negative— between  the  flexibility  of  a  system  and  its  cost,  schedule,  and  performance? 
How  much  does  flexibility  really  cost,  both  in  terms  of  money  and  other  system 
tradeoffs?  Can  a  flexible  system  save  us  money,  and  if  so,  does  this  apply  to  all  phases  of 
the  system  lifecycle  or  only  to  the  procurement  phase  or  the  operations  and 
maintenance  phase?  Similarly,  can  flexibility  provide  a  reduction  in  lifecycle  cost  by 
extending  the  life  of  a  system?  Does  it  allow  us  to  field  a  system  more  rapidly  or 
respond  to  new  threats  more  quickly  and/or  effectively?  Do  we  know  if  optimal 
performance  goals  are  diametric  to  flexibility  goals? 

Finally,  once  we  know  what  flexibility  is,  what  its  value  is,  and  how  we  measure  it,  we 
can  then  focus  on  how  to  achieve  it.  The  key  questions  are:  How  do  we  structure  a 
program  such  that  it  fosters  designflexibility?  Are  there  specific  design  principles  that 
can  be  used  to  instill  flexibility  into  the  system  design?  Moreover,  if  certain  methods  do 
contribute  to  flexibility,  can  this  relationship  be  proved  formally?  Is  there  a  relationship 
between  flexibility  and  other  key  macro -elements  of  the  system,  such  as  architecture 
and  acquisition  strategy?  Do  we  need  to  modify  acquisition  practices  and  procurement 
processes  to  enable  the  implementation  of  flexibility? 

As  our  review  of  the  current  state-of-the-art  will  show,  there  is  little  unifying  theory  or 
guidance  on  best  approaches  to  measure  flexibility,  quantify  value  of  flexibility  in  a 
prospective  systems  acquisition  or  which  MPTs  work  best  in  which  situations. 

Considering  this  major  gap  in  the  current  state-of-the-art  the  primary  focus  of  our 
research  activities  is  in  developing  a  coherent  value  based  definition  of  flexibility  that  is 
based  on  an  analytical  framework  that  is  mathematically  consistent,  domain 
independent  and  applicable  under  varying  information  levels. 


1 .2  Scope  and  Specific  Project  Objectives 

The  primary  aspect  of  valuing  flexibility  emphasized  by  the  sponsors  is  to  focus  on 
flexibility  to  deal  with  foreseeable  sources  of  change,  to  demonstrate  the  value  of 
flexibility  in  quantitative,  monetizable  terms  familiar  to  DoD  organizations  such  as  Cost 
Analysis  and  Program  Evaluation  (CAPE),  and  to  identify  mathematical  tools  for 
determining  the  monetized  value  of  flexibility.  Some  further  focus  areas  in  which  the 
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sponsors  have  expressed  interest  involve  ways  to  improve  flexibility  via  quantifying 
departures  from  requirements  as  the  sources  of  change,  via  determining  enablers  for 
providing  modular  margins  to  improve  flexibility,  and  via  determining  decision  drivers 
for  product  line  scope  decisions. 

The  Phase  l  of  this  project  is  being  performed  over  two  periods.  The  first  7-month 
period,  the  subject  of  this  report,  focused  on  identifying  current  quantitative  MPTs  for 
valuing  flexibility  in  DoD  context,  and  delivering  an  initial  capability  to  value 
investments  that  provide  the  flexibility  to  handle  foreseeable  sources  of  change,  using 
easy-to-understand  monetizable  terms,  such  as  the  effect  of  the  investments  on  total 
cost  of  ownership  or  return  on  investment.  We  conducted  a  critical  evaluation  of  the 
theoretical  foundations  underlying  current  approaches,  the  dimensions  of  flexibility, 
measures  of  flexibility,  value  functions,  and  methods  for  incorporating  flexibility  -  both 
at  the  design  phase  and  the  operational  phase,  to  identify  strengths  and  weaknesses  of 
each  approach.  During  this  period,  we  also  explored  the  development  of  an  analytical 
framework  based  on  sound  mathematical  constructs  that  will  be  refined  in  the  second 
period. 

The  second  6-month  period  will  focus  on  completing  the  development  and  comparative 
evaluation  of  more  advanced  methods  of  valuing  flexibility.  These  analyses  will 
compare  the  various  methods  across  representative  DoD  change  adaptation  situations 
using  a  variety  of  case  studies  of  completed  and  prospective  projects.  This  will  enable 
DoD  projects  to  determine  which  valuation  methods  to  use  in  which  situations.  These 
results  will  also  identify  significant  gaps  between  available  capabilities  and  DoD  needs, 
both  for  valuing  and  fielding  flexibility.  The  gap  analysis  results  will  then  be  used  to 
develop  a  research  roadmap  for  developing  the  needed  capabilities  by  mapping  current 
capabilities  on  future  DoD  operational  requirements. 
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2  Critical  Review  of  the  State-of-the-Art  in 
Valuing  Flexibility 


In  general,  the  terminology  used  in  literature  related  to  flexibility  is  a  quagmire.  In 
most  articles  on  the  subject  of  flexibility,  the  definition  is  not  explicitly  provided,  thus 
leaving  the  reader  to  infer  or  guess  at  its  meaning.  In  many  cases,  the  problem  extends 
to— and  is  exacerbated  by— the  careless  usage  of  many  of  the  other  non-traditional 
system  design  parameters  (i.e.,  the  so-called  “— ilities”)-  For  instance,  the  terms 
“flexibility”  and  “adaptability”  are  often  used  interchangeably,  or  conflated  with 
descriptors  like  agility,  modifiability,  changeability,  universality,  scalability,  etc. 

We  begin  by  considering  how  flexibility  is  used  across  disciplines,  and  then  narrow  our 
attention  to  the  domain  of  particular  interest  to  us:  Designing  a  flexible  defense  system 
or  product.  Once  we  have  carefully  examined  the  various  definitions  of  flexibility,  we 
will  immediately  consider  its  linguistic  doppelganger:  Adaptability.  An  integrated 
examination  of  these  two  terms  should  help  clarify  both.  In  turn,  this  will  help 
contextualize  the  role  of  related  concepts  such  as  agility,  modularity,  interoperability, 
and  robustness,  which  we  will  also  briefly  examine  and  synthesize.  Our  goal  is  to 
formulate  a  precise  definition  of  flexibility  that  can  provide  the  foundation  of 
subsequent  discussion  regarding  flexibility,  and  will  lend  itself  to  quantifying  its  value, 
and  ultimately  help  us  implement  it. 


2.1  Definitions  and  Quantitative  Measures  of  Flexibility 

While  there  is  no  clear  agreement  on  the  definition  of  flexibility,  there  is,  at  least,  broad 
acknowledgment  of  this  fact.  The  lack  of  an  unambiguous,  consistent  definition  for 
flexibility  is  lamented  by  numerous  authors  (e.g.,  [Saleh,  2001;  Saleh,  2003;  Nachtwey, 
2009;  Fitzgerald,  2009;  Bordoloi,  1999]).  And  of  the  “-ilities,”  there  is  reason  to  believe 
that  “flexibility”  is  the  most  exploited.  In  one  study,  the  authors  presented  evidence 
showing  that  the  term  “flexibility”  (and  its  variants)  is  used  in  a  colloquial  sense  much 
more  often  than  other  design  terms  like  “robustness”  and  “optimize”  (and  their 
variants),  concluding  that  the  concept  of  flexibility  lacks  “scholarly  maturity” [Saleh, 
2009].  However,  this  author  also  supports  the  notion  that  the  “concept  of  flexibility  is 
today  where  the  concept  of  quality  was  some  20  years  ago,”  suggesting  that  its  definition 
and  quantification  are  destined  to  mature  [Saleh,  2009]. 

Following  is  a  small  sampling  of  the  types  of  definitional  deviation  that  can  be  found  in 
the  literature  on  flexibility.  Flexibility  is— 
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•  "The  ability  to  change  or  react  with  little  penalty  in  time,  effort,  cost  or 
performance"  [Upton,  1994] 

•  “The  property  of  a  system  that  is  capable  of  undergoing  specified  classes  of 
changes  with  relative  ease”  [Suh,  2007] 

•  “The  best  possible  performance  in  the  face  of  environmental  variability”  [de 
Groote,  1994] 

•  “Adaptable  and  capable  of  change”  [Gustavsson,  1995] 

•  “There  is  a  consensus  in  the  literature  that  flexibility,  sometimes  called 
'versatility’,  means  adaptability  to  changes”  [Barad,  1997] 

Many  of  the  definitions  found  in  literature  are  too  broad  and  self-referential  to  be  of  use 
in  quantitative  assessments  and  decision  making.  Also,  the  tendency  to  use 
“adaptability”  as  a  means  of  explaining  “flexibility”  occurs  frequently,  and  serves  to 
obfuscate  the  matter  further. 

Several  researchers  have  attempted  to  define  specific  types  of  flexibility.  Below  is  a  list 
of  some  of  these  non  domain-specific  flexibility  subtypes,  along  with  one  sample 
definition  from  the  literature. 

•  Organizational  Flexibility:  “The  ease  with  which  the  organization's  structures  and 
processes  can  be  changed”  [Nelson,  1997] 

•  Process  Flexibility:  “The  ability  of  people  to  make  changes  to  the  technology 
using  management  processes  that  support  business  process  changes”  [Nelson, 
1997] 

•  Structural  Flexibility:  “The  capability  of  the  design  and  organization  of  a 
business  process  changes”  [Nelson,  1997] 

•  Technology  Flexibility:  “The  ability  to  adapt  to  both  incremental  and 
revolutionary  change  in  the  business  or  business  process  with  minimal  penalty  to 
current  time,  effort,  cost,  or  performance.”  [Nelson,  1997] 

•  Management  Flexibility:  “The  ability  of  management  to  affect  the  course  of  a 
project  by  acting  in  response  to  the  resolution  of  market  uncertainty  over  time” 
[Saleh,  2003] 

•  Product  Flexibility:  ““Product  flexibility  can  be  defined  as  the  degree  of 
responsiveness  (or  adaptability)  for  any  future  change  in  a  product  design”  {{180 
Rajan,Palani,P.K.  2003}}. 

•  Decision  Flexibility:  “The  number  of  remaining  alternatives  after  a  first 
commitment  is  made”  [Saleh,  2009] 

•  Operational  Flexibility:  “Attributes  of  the  system  that  emerge  in  the  face  of 
unanticipated  changes”  [Nilchiani,  2006] 

•  Development  Flexibility:  “Can  be  expressed  as  a  function  of  the  incremental 
economic  cost  of  modifying  a  product  as  a  response  to  changes  that  are  external 
(e.g.,  a  change  in  customer  needs)  or  internal  (e.g.,  discovering  a  better  technical 
solution)  to  the  development  process”  [  Thomke,  1998] 
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•  Design  Flexibility:  “Implies  that  the  system  has  been  designed  with  certain 
characteristics”  which  “may  not  be  necessary  or  justified  in  the  context  of 
optimising  the  system  for  the  immediate  mission  (or  set  of  requirements)  ... 
However,  these  characteristics  will  allow  the  system  to  be  easily  modified  should 
these  requirements  change  after  it  has  been  fielded.”  [Saleh,  2009] 

•  Design  Process  Flexibility:  “Denotes  a  willingness  and  an  ability  to 

accommodate  requirement  changes  during  the  design  process  (i.e.  before  the 
system  is  fielded),  and  it  characterises  either  the  interaction  between  the 
customers  and  designers  ...  or  between  different  teams  of  designers  working  on 
separate  subsystems  of  a  complex  engineering  design”  [Saleh,  2009] 

Scouring  the  literature  on  flexibility,  we  find  that  all  remotely  viable  definitions  do  share 
the  common  element  of  change.  However,  not  all  definitions  agree  that  it  must  be  the 
system  that  undergoes  change  in  order  for  it  to  be  deemed  flexible.  Moreover,  the 
nature  of  the  change,  its  source,  when  it  occurs,  and  how  it  occurs  are  all  potentially 
differentiating  elements  of  the  published  definitions.  We  do  find,  though,  that  the 
elements  of  the  flexibility  definitions  can  generally  be  wholly  described  by  the  answers 
to  one  or  more  (usually  more)  of  the  following  five  questions: 

1.  Will  the  System  Change?  This  refers  to  whether  the  system  under  consideration 
must  change  in  some  manner  to  be  considered  flexible.  The  alternative  is  that 
the  system  does  not  necessarily  change  itself,  but  rather  “accommodates”  the 
instigating  change. 

2.  What  Measure(s)  of  Change  Efficiency  Applies?  This  aspect  of  the  definition 
provides  a  description  of  how  efficient  the  system  change  is,  relative  to  resources 
like  time  and  money. 

3.  What  is  the  Source  of  the  Change?  Asking  this  tells  us  where  the  instigating 
change  force  is  relative  to  the  system,  i.e.,  internal  or  external. 

4.  Is  the  Change  Foreseeable?  Aims  to  capture  whether  the  potential  change  is  one 
that  can  be  anticipated. 

5.  When  may  the  Change  Occur?  This  question  relates  to  when  in  the  system’s 
lifecycle  it  may  be  exposed  to  the  change,  with  the  delineation  being  whether  the 
change  occurs  before  or  after  the  system  is  fielded. 

Table  1  provides  a  summary  of  all  the  definitions  of  flexibility  binned  according  to  the 
five-question  template.  While  adaptability  may  be  the  term  most  often  confused  with 
flexibility,  it  is  certainly  not  the  only  one.  The  relationship  between  flexibility  and 
robustness  can  be  muddled  as  well.  One  scholar  asserts  that  “robustness”  and 
“adaptability”  are  the  two  design  components  that  enable  a  system  to  have  flexible 
performance  [Olewnik,  2001].  Table  2  shows  a  similar  summary  of  the  definitions  of 
adaptability,  and  the  summary  of  the  robustness  definitions  is  provided  in  Table  3. 
Several  other  “ilities”  and  their  relationship  with  flexibility  are  discussed  in  detail  in  the 
Appendix. 
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Before  the  value  of  flexibility  can  be  ascertained,  one  needs  to  quantify  flexibility  or 
develop  metrics  that  can  be  then  mapped  on  the  value  functions.  Several  measures  have 
been  proposed  in  literature  to  quantify  flexibility  based  on  decision  theoretic  principles, 
ranging  from  counting  number  of  choices,  options,  or  systems  states  affected,  graph 
theoretic  measures,  to  Shannon-Weaver  type  entropy  measures,  and  uncertainty  and 
dispersion  measures  [Buzacott,  1982;  Sethi  and  Sethi,  1990;  Deshmukh,  et  al.  1998]. 
Other  measures  of  flexibility  are  based  on  the  methods  used  to  incorporate  flexibility  in 
systems,  such  as  interoperability  and  modularity  [Jacques  and  Colombi,  2009;  Ford  and 
Colombi,  2009;  Stryker  and  Jacques,  2010].  Further  methods  include  real  options, 
decision  theory,  insurance  analysis,  risk  analysis,  attribute  tradeoff  analysis,  cost  of 
delay  analysis,  and  product  line  ROI  analysis.  They  tend  to  work  better  for  monetized 
commercial  applications  than  for  DoD  measures-of-effectiveness. 
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FLEXIBILITY 

System  Change? 

Measure(s)  of 
Change  Efficiency 

Source  of 
Change 

Foreseeable? 

When  Change 

Occurs 

Other 

SOURCE 

Yes 

Not 

Necessarily 

Quickly 

Cost- 

Effectively 

Int 

Ext 

Reqmnt 

Known 

Unk 

Prior  to 
Fielding 

After 

Fielding 

Related  Terms 

Thomke,  1997 

"modifying  a 
product" 

"incremental  cost 

and  time" 

X 

X 

Roser,  1999 

"change 

performance" 

"minor  time  and 

costs" 

Schulz,  1999 

"system  to  be 
changed" 

"changed  easily" 

Component  of  Changeability 

Bordoloi,  1999 

"change  states" 

Efficiency  and  Adaptability 

Olewnik,  2001 

"changes  in 
configuration" 

"real¬ 

time" 

X 

X 

X 

X 

Robustness  and  Adaptability  are 
modes  of  Flexibility 

Palani,  2003 

"design 

changes" 

"ease"  of  change 

Nilchiani,  2003 

"increasing  con¬ 
trol  capacity" 

X 

X 

Banerjee,  2004 

"support  new 

functions" 

Nilchiani,  2006 

"ability  to 
respond" 

"timely" 

"cost- 

effective" 

X 

X 

Qureshi,  2006 

"responsive¬ 

ness" 

"degree  of 
responsiveness" 

Keese,  2007 

"redesign" 

"quickly" 

"inexpen¬ 

sively" 

X 

X 

Ross,  2008 

"alterations" 

X 

Shah,  2008 

"ability  to 
change" 

X 

Adaptability  and  Flexibility 
comprise  Changeability 

Sivanthi,  2008 

"accommodate 
new  reqmnts" 

X 

Fitzgerald,  2009 

"ability  to 
adapt" 

X 

X 

Viscito,  2009 

X 

Brown,  2009 

"change  on 

demand" 

Scalability,  Evolvability, 
Maintainability,  and  Adaptability 

Nachtwey,  2009 

"facilitates  an 
adaptation" 

"effective  and 

efficient" 

X 

Flexible  system  must  be  Robust 
and  Adaptable 

Saleh,  2009 

"modified" 

"timely" 

"cost- 

effective" 

X 

X 

X 

X 

Lafleur,  2010 

"modify  a 
system" 

"easily  modify" 

X 

X 

X 

Merriam 

Webster,  2010 

"capability  to 
adapt" 

"ready" 

X 

Table  1:  Summary  of  Flexibility  Definitions 
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ADAPTABILITY 

System  Change? 

Source  of 
Change 

Foreseeable? 

When  Change 

Occurs 

Other 

SOURCE 

Yes 

Not 

Necessarily 

Int 

Ext 

Reqmnt 

Known 

Unk 

Real¬ 

time 

Offline 

Related  Terms 

Schulz,  1999 

"adapt  itself 

X 

(in  operations) 

Robustness  is  a  prerequisite 
of  Adaptability 

Bordoloi,  1999 

"change 

states" 

Olewnik,  2001 

"enhance 

performance" 

X 

X 

X 

X 

X 

Robustness  and  Adaptability 
are  modes  of  Flexibility 

Haubelt,  2002 

"react  to 
changes" 

X 

X 

Chung,  2004 

"accommodate 

changes" 

X 

Ross,  2008 

"alterations" 

X 

Shah,  2008 

"ability  to 
change" 

X 

Brown,  2008 

"ability  to 
reconfigure" 

X 

(in  operations) 

Reconfigurability  or 
Versatility 

Table  2:  Summary  of  Adaptability  Definitions 
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ROBUSTNESS 

System  Change? 

Source  of 
Change 

Foreseeable? 

When  Change 

Occurs 

Other 

SOURCE 

Yes 

No 

Int 

Ext 

Known 

Unk 

Prior  to 
Fielding 

After 

Fielding 

Related  Terms 

Phadke,  1989 

minimize  effect  on 
performance 

X 

X 

X 

Schulz,  1999 

"deliver  their  intended 
functionality" 

X 

X 

Olewnik,  2001 

"accommodating 

change" 

X 

X 

X 

Robustness  and  Adaptability  are  modes  of 
Flexibility 

Carlson,  2002 

maintain  desired 
system  characteristics 

X 

X 

Saleh,  2003 

"fixed  set  of  reqmnts" 

X 

X 

X 

X 

Banerjee,  2004 

"fixed  behavior" 

X 

X 

Ross,  2006 

"remain  constant" 

X 

Ross,  2008 

"remain  constant" 

X 

X 

Shah,  2008 

"continue  delivering 
value" 

X 

Saleh,  2009 

"fixed  set  of  reqmnts" 

X 

X 

Resistance  to  Immunity  to  Change 

Brown,  2009 

"maintain 

functionality" 

X 

Consists  of  Reliability,  Survivability, 
Resilience  to  Fragility,  and  FauItTolerance 

Merriam- 

Webster,  2010 

"performing  without 
failure" 

X 

X 

Table  3:  Summary  of  Robustness  Definitions 
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2.2  Value  of  Flexibility 

With  a  sense  of  what  flexibility  is,  and  how  it  relates  to  the  associated  system 
terminology,  we  now  consider  the  value  of  flexibility  and  why  it  is  needed.  Flexibility  is 
widely  regarded  as  a  beneficial  system  characteristic  [Fitzgerald,  2009;  Thomke,  1998; 
Bordoloi,  1999;  Kumar,  1999;  Saleh,  2001;  Thomke,  1997;  Schulz,  1999].  The  basic 
catalyst  for  flexibility  is  system  responsiveness.  Its  value  is  couched  in  terms  of  being 
able  to  increase  the  likelihood  of  responding  to  changes  more  rapidly  and  at  lower  cost. 
Following  qualitative  statements  are  found  in  literature  on  the  dimensions  of  value  of 
flexibility: 

•  “is  very  critical  in  addressing  changing  customer  needs”  [Banerjee,  2004] 

•  “[is]  essential  to  lowering  risk  and  reducing  the  cost  and  length  of  product 
development”  [General  Accounting  Office  2001] 

•  “is  of  exceptional  value;  it  allows  firms  to  invest  less  time  and  fewer  resources  on 
activities  aimed  at  minimizing  risk”  [Thomke,  1997] 

•  is  suitable  for  “shorter  life-cycles  of  the  products  and  technologies”  and  “shorter 
delivery  times”  [Nilchiani,  2003] 

•  “plays  a  significant  role  in  responding  faster  to  customer  feedbacks  by  allowing 
quicker  updates  in  the  products”  [Rajan,  2003] 

•  is  needed  “when  the  system’s  technology  base  evolves  on  time  scales  considerably 
shorter  than  the  system’s  design  lifetime,  thus  requiring  a  solution  for  mitigating 
risks  associated  with  technology  obsolescence.”  [Saleh,  2009] 

•  provides  the  benefits  of  “technology  insertion  throughout  the  entire  system  life- 
cycle  ...  upgrade  opportunities  and  the  ease  of  customization  ...  rapid 
responsiveness  to  emerging  and  changing  markets  ...  reduced  life  cycle  cost” 
[Schulz,  1999] 

•  “allows  firms  to  invest  less  time  and  fewer  resources  on  activities  aimed  at 
minimizing  risk”  [Thomke,  1997] 

Flexibility  is  also  described  as  providing  the  opportunity  “to  make  (late)  design  changes 
that  lead  to  better  design  solutions  with  respect  to  customer  needs”  and  to  “pursue  a 
more  efficient  development  strategy  that  can  tolerate  a  higher  risk  of  design  changes 
[Thomke,  1997].  Flexibility  also  contributes  to  “achieving  higher  levels  of  performance” 
[Rajan,  2003],  “increased  customization”  [Nilchiani,  2003],  and  “superior  system 
capabilities”  [Schulz,  1999],  and  can  “eliminate  performance  tradeoffs”  [Olewnik, 
2006].  Moreover,  flexibility  may  even  extend  the  life  of  the  system,  being  described  as 
an  “antidote  to  obsolescence”  due  to  its  ability  to  keep  pace  with  changes  [Saleh,  2009]. 

Despite  the  widespread  affirmation  of  flexibility  and  its  lauded  ability  to  improve  system 
responsiveness,  no  sources  could  be  found  that  provided  any  sort  of  quantification  of  the 
cited  benefit.  This  is,  perhaps,  to  be  expected  given  that  the  definitions  of  flexibility  are 
so  divergent,  and  generally  do  not  allow  for  such  a  question  to  have  meaning.  In  other 

Contract  Number:  H98230-08-D-01 71  DO  02,  TO  02  RT018 

Report  No.  SERC-2010-TR-010 
09/25/2010 


UNCLASSIFIED 

19 


UNCLASSIFIED 


words,  how  can  we  say  how  much  time  and  money  was  saved  by  our  flexible  design  if  we 
can’t  even  say  how  flexible  the  design  was? 

For  now,  it’s  important  to  understand  under  what  conditions  flexibility  has  value.  Note 
that  most  value  judgments  referred  to  indefinite  concepts  like  change,  risk,  and 
evolution.  The  common  element  to  all  assessments  of  flexibility’s  value  lies  in  the 
principle  of  uncertainty.  Several  sources  made  this  observation,  and  all  agreed  that  the 
value  of  flexibility  is  greater  under  uncertainty  [Thomke,  1997;  Gershenson,  2003; 
Rajan,  2005;  Saleh,  2009;  Shah,  2008],  and  that  the  greater  the  uncertainty,  the  greater 
the  value  of  flexibility  [Suh,  2007;  Hayes,  2006;  Krishnan,.  2002;  Haubelt,  2002; 
Hutchinson,  1989].  Krishnan  [2002]  discusses  the  value  of  flexibility  in  the  face  of 
technology  uncertainty.  Nilchiani  [2003]  and  Saleh  [2001]  go  a  step  further  and  insist 
that  flexibility  is  precisely  what  is  required  in  order  to  cope  with  uncertainty.  Hastings 
[2006]  conceives  of  a  four-category  design  framework  intended  to  mitigate  uncertainty 
in  complex  systems.  The  framework  begins  with  sources  of  “uncertainty,”  passes 
through  “risks/opportunities”  and  “mitigations,”  culminating  in  possible  “outcomes” 
like  reliability,  robustness,  versatility,  flexibility,  and  interoperability.  For  Hastings, 
flexibility  is  one  of  several  system  attributes  necessary  to  contend  with  uncertainty. 

The  reasons  why  flexibility  is  vital  in  the  face  of  uncertainty  should  be  clear.  System 
development  is  a  dynamic  endeavor,  typically  consisting  of  a  host  of  changing  variables, 
including  market  volatility,  personnel  turnover,  requirement  creep,  new  laws  and 
regulations,  budget  fluctuations,  technological  breakthroughs,  test  failures,  etc.  A 
standard  mitigation  technique  to  deal  with  uncertainty  is  to  postpone  difficult  decisions, 
and  keep  the  maximum  number  of  options  open  for  as  long  as  possible.  Flexibility  can 
allow  a  program  to  do  exactly  that,  thereby  improving  the  likelihood  of  a  more  optimal 
decision  [Brown,  2008;  Gershenson,  2003]. 

The  Cost  of  Flexibility:  The  central  maxim  of  economic  theory,  the  no-free-lunch 
theorem,  is  always  applicable  to  systems  engineering,  conveying  the  important  point 
that  it  is  generally  necessary  to  tradeoff  certain  goals  against  others.  As  valuable  as 
flexibility  may  be,  there  is  bound  to  be  some  kind  of  tradeoff.  Identifying  and 
characterizing  this  tradeoff  is  vital  if  we  wish  to  invest  in  flexibility;  however,  there  is 
surprisingly  little  discussion  in  the  literature  as  to  the  nature  of  this  tradeoff. 

For  instance,  a  metric  does  not  appear  to  exist  for  determining  how  much  flexibility 
costs,  though  one  source  implies  that  flexibility  (and  modularity)  are  negatively 
correlated  to  performance  Ulrich  [1999]  and  another  claims  that  flexibility  often  leads  to 
over-capacities,  with  the  inference  being  that  resources  are  squandered  [Nachtwey, 
2009].  Others,  such  as  Olewnik  [2001],  Nilchiani  [2006],  and  Eckert  [2004]  also  warn 
that  there  are  “tradeoffs”  when  pursuing  flexibility,  and  that  there  is  likely  to  be  a 
potential  “upfront  cost.”  Nachtwey  [2009]  conceives  of  this  tradeoff  “as  an  insurance 
premium  which  is  paid  at  present  in  order  to  have  a  possible  advantage  in  future.”  Saleh 
[2009]  recognizes  that  there  is  bound  to  be  some  cost/performance  penalty  associated 
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with  flexibility  and  proposes  a  series  of  questions  that  warrant  further  investigation, 
e.g.,  “Are  flexibility  and  optimization  competing  paradigms  for  system  design?  Is 
flexibility  obtained  at  the  cost  of  some  performance  penalty,  and  do  flexible 
designs/systems  exhibit  a  performance  gap  compared  with  the  alternative  optimized 
designs/systems?”  It  does  not  appear  that  any  researcher  has  taken  on  these  questions. 
The  closest  is  a  single  case  study  conducted  in  the  manufacturing  domain  where  the 
researcher  found  that  the  flexible  design  cost  34  percent  more  to  implement  at  first,  “but 
will  incur  significantly  lower  switching  costs  when  the  vehicle  design  changes”  [Suh, 
2007].  This  researcher  adds,  however,  that,  “the  best  flexible  designs  may  not  increase 
up-front  investment  at  all.” 

Another  study,  commissioned  by  the  Air  Force  Office  of  Scientific  Research  focuses  on 
the  idea  of  decision  flexibility.  In  attempting  to  answer  the  question  of  how  much 
flexibility  is  enough,  they  find  that  “Lots  of  flexibility  is  not  significantly  better  than  a 
little  flexibility.  The  impact  of  flexibility  on  performance  exhibits  strong  diminishing 
returns,  with  most  of  the  benefit  realized  with  relatively  limited  flexibility.”  [Hayes, 
2006] 

If  it’s  true  that  flexibility  is  highly  valuable,  and  it’s  also  true  that  we  can’t  get  something 
for  nothing,  we  must  conclude  that  we  must  sacrifice  something  for  flexibility. 
Performance— particularly  optimized  performance— would  appear  to  be  the  likely 
candidate,  but  the  evidence  is  lacking  at  this  time.  We  also  note  that  there  is— not 
surprisingly— no  discussion  regarding  the  additional  challenge  of  implementing  design 
flexibility  for  defense  acquisition,  where  all  resources  expended  must  be  justified  in 
terms  of  a  validated  requirement.  Investing  additional  resources  for  a  design  that 
exceeds  an  established  requirement  or  may  be  well  suited  to  accommodate  a  non¬ 
existing  requirement  is  clearly  at  odds  with  defense  acquisition  strategy.  Similarly, 
suggesting  that  resources  should  be  spent  on  a  nebulous  concept  of  goodness  like” 
flexibility”  would  likely  fair  no  better. 

The  Need  for  Flexibility:  Just  because  something  has  value  doesn’t  necessarily  mean 
that  it’s  worth  pursuing.  Value  is  not  an  absolute  concept.  For  instance,  to  the 
individual  stranded  on  a  desert  island,  a  million  dollars  is  bound  to  have  less  value  than 
a  shortwave  radio.  So  the  more  important  question— and  the  one  that  represents  the 
central  motivation  for  this  study— is  should  we  seek  to  implement  flexibility  in  defense 
acquisitions.  Based  on  the  academic  literature,  as  well  as  various  government 
publications,  the  answer  is  a  resounding  “yes.” 

Consider  the  following  list  of  potential  problems  that  may  be  resolved  or  mitigated 
through  greater  flexibility: 

•  “Complex  DoD  systems  tend  to  be  designed  to  deliver  optimal  performance  within  a 
narrow  set  of  initial  requirements  and  operating  conditions  at  the  time  of  design. 
This  usually  results  in  the  delivery  of  point-solution  systems  that  fail  to  meet 
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emergent  requirements  throughout  their  lifecycles,  that  cannot  easily  adapt  to  new 
threats,  that  too  rapidly  become  technologically  obsolete,  or  that  cannot  provide 
quick  responses  to  changes  in  mission  and  operating  conditions.”  (OSD:  RT-18  Task 
Description) 

•  “The  customer  often  wants  a  system  to  have  ilities,  but  is  not  willing  (or  does  not 
know  how)  to  pay  for  them.  Additionally,  even  though  customers  often  request  these 
types  of  ilities  when  acquiring  systems,  it  was  unclear  how  to  specify,  evaluate,  and 
validate  these  ilities  requirements  for  systems.”  [Ross,  2008] 

•  “A  framework  that  allows  for  consideration  of  ilities  during  conceptual  design, 
including  concrete  specification  of  how  the  ilities  are  defined,  can  be  quantified  and 
relate  to  perceived  value,  would  benefit  both  military  and  industry”  [Ross,  2008] 

•  “Modern  systems  engineering  requires  operating  in  an  ever  changing  environment 
within  which  systems  often  need  to  adapt  to  maintain  value  delivery  and  to  take 
advantage  of  emergent  opportunities.”  [Shah,  2008] 

•  “There  is  a  need  for  a  comprehensive  framework  that  allows  decision-makers  to 
measure  the  value  of  flexible  systems  design  in  its  different  dimensions.”  [Nilchiani, 
2006] 

•  For  2008  DOD  programs,  “research  and  development  costs  are  now  42  percent 
higher  than  originally  estimated  and  the  average  delay  in  delivering  initial 
capabilities  has  increased  to  22  months. ’’[General  Accounting  Office  2009] 

If  flexibility  could  be  characterized  and  implemented  correctly,  the  benefit  to 
government  acquisition  programs  would  be  almost  beyond  measure.  The  current 
lengthy  development  timelines  and  rigid  acquisition  processes  could  give  way  to  a  new 
paradigm  of  quick-response  modifications  to  existing  designs  or  fielded  products.  The 
ability  to  respond  more  rapidly,  in  turn,  generates  a  self-reinforcing  feedback  loop  that 
would  reduce  the  likelihood  of  a  system  being  exposed  to  additional  sources  of  change, 
i.e.,  evolving  customer  requirements,  new  strategic  direction,  budget  shortfalls,  and 
technology  obsolescence.  Tools  to  characterize,  measure,  and  implement  flexible  design 
solutions  would  be  of  value  during  virtually  all  phases  of  system  development,  including 
concept  definition,  analysis  of  alternatives,  source  selection,  and  all  program  milestones 
(e.g.,  SRR,  PDR,  CDR).  The  ability  to  design  systems  to  achieve  flexibility  could  also 
change  the  calculus  of  how  we  determine  when  to  start  a  new  program  vice  modifying  an 
existing  one.  Flexibility  could  even  help  reinvent  the  concept  of  risk  management,  and 
allow  programs  to  see  risk  as  just  one  aspect  of  uncertainty  that  has  both  negative  and 
positive  outcomes  that  contribute  to  an  overall  determination  of  value  [Brown,  2009; 
Collopy,  2003;  Nilchiani,  2006]. 


2.3  Quantitative  Methods  for  Valuing  Flexibility 

If  we  lack  consensus  on  what  flexibility  is,  then  it  is  to  be  expected  that  quantifying  its 
value  will  be  that  much  harder  [Bordoloi,  1999].  And  while  success  remains  elusive 
[Brown,  2008;  Baykasoglu,  2009],  there  have  been  a  surprising  number  of  attempts. 
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Unfortunately,  most  of  these  attempts  only  tackle  part  of  the  problem  or  are  highly 
restricted  in  their  applicability  [Rajan,  2005],  such  that  we  still  lack  a  robust,  effective 
method  to  formally  quantify  the  value  of  flexibility  [Saleh,  2003].  Nevertheless,  it  is 
worthwhile  to  review  these  efforts  so  we  can  better  appreciate  which  ones  hold  promise, 
and  which  ones  do  not. 

Within  the  topic  of  valuing  flexibility  in  the  foreseeable-change  area,  there  are  also  two 
primary  cases:  flexibility  for  a  single  system;  and  flexibility  for  a  portfolio  or  product  line 
of  similar  systems.  The  Parnas  strategy  applies  to  both,  but  in  different  ways.  For 
single  systems,  one  can  determine  or  estimate  the  cost  of  creating  and  using  a  modular 
design  (including  its  effects  on  other  system  attributes  such  as  performance  and  time- 
to-deliver).  One  can  then  use  real-options  approaches  based  on  estimates  of  the 
probabilities  and  timing  of  sources  of  change,  and  the  cost-of-change  savings  of  having 
the  modular  design,  to  estimate  the  value  of  the  modular-design  approach  and  return  on 
investment  in  the  modular  design  [Sullivan  et  al.,  1999;  2001].  Similar  work  has  been 
done  by  [Mun,  2002;  2005]  and  by  Baldwin-Clark  [2000]  and  DeNeufville  [2001; 
2003].  An  important  factor  in  using  existing  real  options  results  in  developing  a 
framework  for  valuing  flexibility  is  that  some  of  the  basic  assumptions  underlying  the 
results,  such  as  fluidity  of  market  and  non-correlated  events,  do  not  apply  in  the  system 
flexibility  valuation.  [Ball,  2007]  have  developed  methods  to  determine  the  value  of  an 
option  when  the  underlying  assumptions  do  not  hold,  and  the  resulting  uncertainty  may 
be  due  to  a  smart  adversary,  not  just  randomness  in  the  environment. 

An  attractive  approach  for  both  valuing  and  determining  flexible  designs  involves 
various  forms  of  search  and  optimization  of  the  design  trade  space  of  key  design 
parameters.  This  has  been  successfully  applied  in  the  DARPA  F6  satellite  program 
[Brown-Eremenko,  Collopy,  2006;  2007;  2009]  and  in  the  MIT  Lean  Aerospace 
Initiative  [Ross,  Ross-Hastings;  2006].  Another  candidate  approach  involves 
comparative  Monte  Carlo  analyses  of  change  sequences  with  and  without 
modularization  using  systems  dynamics  techniques  [Madachy,  2008].  Some  further 
attractive  candidate  approaches  involve  monetizing  the  cost  of  delay  in  system  fielding 
due  to  inadequate  design  for  flexibility,  through  various  value  estimating  relationships 
relating  time  of  delivery  to  organizational  value  [Huang-Boehm,  2006];  the  use  of 
commercially-equivalent  value  streams  [Mun,  2005],  and  insurance-based  valuation 
approaches  [Raz-Shaw,  2001]. 

There  are  several  parallels  between  valuing  flexibility  and  valuing  insurance  or  warranty 
payments.  Both  flexibility  and  insurance  are  hedges  against  future  uncertainty,  whether 
in  system  performance  or  intended  use.  Lian  and  Deshmukh  [2009]  have  shown  that 
even  moderate  amounts  of  flexibility  in  supply  contracts  can  have  significant  impact  on 
the  overall  effectiveness  of  complex  supply  chains.  Current  actuarial  and  warranty 
research  will  be  examined  to  determine  the  appropriate  mapping  between  methods  for 
optimal  pricing  of  risk  in  those  areas  and  valuing  flexibility  in  system  acquisition  setting. 
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For  a  portfolio  or  product  line  of  similar  systems,  modularizing  around  foreseeable 
sources  of  change  involves  analyzing  the  commonalities  and  variabilities  across  the 
product  line,  and  developing  reusable  components  for  the  common  features  and  plug-in 
interfaces  for  the  variable  features.  A  good  example  is  the  Hewlett  Packard  product  line 
experience.  For  a  product  line  of  medical  and  engineering  measurement  devices,  they 
reduced  their  time  to  delivery  from  48  months  to  12-14  months  by  investing  in  two 
projects  that  took  57  and  61  months  to  develop  products  whose  components  could 
largely  be  reused  in  subsequent  products  [Griss,  1995].  Delayed  differentiation  and 
mass  customization  are  often  effective  design  strategies  in  commercial  product  lines. 

Valuing  software  product  line  reuse  has  been  extensively  analyzed.  Poulin  [1997] 
summarizes  the  results  of  a  dozen  reuse  studies,  and  concludes  in  general  that 
successful  reuse  requires  an  added  50%  effort  in  developing  and  certifying  reusable 
components,  and  that  reuse  without  modification  costs  only  20%  of  the  cost  of 
redeveloping  components,  and  that  reuse  investments  pay  off  for  the  development  of 
product  lines  of  at  least  3  similar  products.  Reifer  [1997;  2002]  reports  similar  results. 
The  Selby  [1988]  analysis  of  about  3000  NASA  software  modules  found  that  reuse  with 
modification  of  the  components  was  considerably  more  expensive,  adding  20-50%  to 
the  cost  of  reuse.  These  results  were  calibrated  to  161  projects  in  the  COCOMO  II  cost 
model  [Boehm  et  al,  2000],  and  extended  the  Constructive  Product  Line  Investment 
Model  (COPLIMO)  [Boehm  et  al.,  2004],  which  calculates  the  return  on  investment 
across  a  multi-year  maintenance  period,  and  finds  even  greater  rates  of  return  when 
considering  the  total  cost  of  ownership  of  the  product  line  versus  developing  and 
maintaining  separate  products.  Total  cost  of  ownership  appears  to  be  a  convincing 
monetized  way  of  justifying  investments  in  flexibility  via  product  line  reuse  for 
organizations  such  as  DoDCAPE. 

Similar  product  line  reuse  savings  have  been  found  for  systems  engineering  in  the 
calibration  of  the  COSYSMO  cost  estimation  model  [Valerdi,  2005;  Fortune,  2009].  The 
use  of  portfolio  risk  management  [Mun,  2003;  2005]  provides  another  strong 
perspective  on  valuing  a  product  line  portfolio,  which  also  address  the  sponsors’  interest 
in  determining  decision  drivers  for  product  line  scope  decisions.  A  list  of  25  critical 
decision  drivers  and  success  factors  for  product  lines  were  developed  in  the  DARPA 
Domain-Specific  Software  Architectures  program  [Boehm-Scherlis,  1992].  These  were 
grouped  into  the  categories  of  architecture  determination,  architecture/component 
description,  component  construction,  component  composition/assemble,  and 
component  interchange.  Elaborations  and  case  studies  on  these  and  other  scope 
decision  drivers  are  well  covered  in  the  book  Software  Product  Lines  [Clements- 
Northrop,  2002]. 

Most  approaches  for  valuing  flexibility  depend  on  good  estimation  of  uncertainty. 
However,  estimating  and  characterizing  uncertainty,  even  for  foreseeable  sources  of 
chance,  is  difficult,  especially  in  systems  involving  new  technologies.  [Wortman,  2009; 
Wortman  et  al.  1994]  have  developed  approaches  based  on  risk  analysis  in  high  stakes 
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wagering  games  to  determine  how  much  information  is  needed  to  develop  a  good- 
enough  characterization  of  uncertainty  that  can  be  used  to  quantify  the  value  added  by 
given  measure  of  flexibility  given  known  areas  of  potential  change. 


2.4  Methods,  Processes  and  Tools  for  Incorporating  Flexibility 
in  Systems 

Let’s  assume  we  know  what  flexibility  is,  and  we  deem  it  be  valuable.  Let’s  further 
assume  that  we  can  measure  its  value.  Then  “all”  that  remains  is  the  task  of 
implementing  it.  Fitzgerald  [2009]  states,  “how  [flexibility]  can  be  achieved  is  less 
obvious.”  Saleh  [2009]  is  particularly  pessimistic:  “there  is  not  yet  a  coherent  set  of 
results  that  demonstrates  how  to  embed  flexibility  in  the  design  of  engineering  systems.” 

The  current  literature  on  engineering  design  does  not  provide  a  formal  approach 
towards  designing  products  for  flexibility.  The  reason  for  this  is  related  to  the  inherent 
properties  of  the  mechanical  design  process.  Because  the  overall  function  in  a 
mechanical  system  is  achieved  through  the  interactions  among  many  subsystems  and 
components  which  are  often  useful  only  in  their  exact  configuration.  Therefore,  any 
structural  or  functional  modifications  are  very  difficult  to  make  in  order  to  adapt  the 
system  to  the  changed  conditions.  As  a  result  of  this  typical  mechanical  systems  are 
designed  for  a  specific  operational  mode. 

There  are,  however  several  techniques  for  enhancing  a  design  with  respect  to  known 
flexibility.  These  techniques  include  modular  design,  product  family  development,  and 
platform  design.  The  underlying  principle  in  these  techniques  is  the  segmentation  of  a 
product  [Hashemian,  2005].  Since  structural  connectivity  is  an  inherent  property  of 
mechanical  systems,  in  order  to  reduce  propagation  of  changes  the  choice  is  limited  to 
two  categories:  use  of  alternative  technologies,  and  segmentation  of  the  structure.  The 
first  category  basically  avoids  the  use  of  solid  components  and  their  associated  spatial 
constraints,  and  replaces  them  with  hydraulic,  electronic  and  software  systems.  The 
second  category  is  based  on  the  premise  that  in  a  modular  structure  modifications  are 
likely  to  be  confined  within  segments  and  are  less  prone  to  propagating  into  other 
segments. 

The  proposed  solution  approach  follows  the  second  category  in  order  to  make  a  product 
adaptable.  The  adaptability  has  to  be  build-in  during  the  design  stage.  The  methodology 
for  an  ’Adaptable  Design’  (AD)  is  to  take  advantage  of  available  ’forecast’  information 
and  design  for  predetermined  adaptations  first.  This  adaptability  is  categorized  as 
Specific  AD;  these  methods  design  products  for  versatility,  upgrading,  variety,  and 
customization.  The  second  step  then  is  to  design  for  General  AD,  which  is  the  design  for 
’unforeseen’  changes.  General  AD  involves  fundamental  research  in  design  theory  and 
methodology  in  order  to  develop  practical  design  methods  and  guidelines.  Hashemian 
proposes  the  subordination  of  a  system  to  a  rational  functional  structure  as  an  approach 
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for  increasing  general  adaptability.  Meaning,  the  general  design  system  consists  of  a 
hierarchical  assembly  of  autonomous  functional  modules. 

A  similar  model  is  proposed  by  Gu  et  al.  [2004].  They  argue  that  the  most  effective  way 
to  increase  the  general  adaptability  of  a  product  is  through  the  use  of  a  ’segregated 
architecture’  (clustering  and  modularization).  This  will  prevent  the  change  in  one  place 
of  the  product  to  propagate  into  the  rest  of  the  product.  They  formulate  general  design 
guidelines  based  on  functions  independence.  In  addition  to  these  guidelines  they  also 
develop  a  measure  of  adaptability  for  products.  The  developed  adaptability  factor 
represents  the  normalized  savings  achieved  by  adaptation  versus  dedicated  product. 
This  measure  is  applicable  to  new  as  well  as  existing  designs. 

The  work  by  Willems  et  al.  [2004]  focuses  on  the  research  question  of  how 
adaptable/flexible  a  design  is.  Given  a  product  design,  new  or  existing,  they  develop  a 
dedicated  methodology  for  assessing  the  adaptability  of  a  product  (MAAP).  Similar  to 
the  work  by  Hashemian  and  Gu  is  the  work  by  Chmarra  et  al.  [2008].  However,  their 
focus  is  mainly  on  the  specific  adaptability,  which  is  related  to  predictable/anticipated 
events.  They  also  propose  the  use  of  modular  design  to  achieve  adaptability  without 
change  propagation  throughout  the  entire  product.  As  stated  before,  prediction  of  such 
change  and  its  propagation  would  be  of  great  help  when  (re)designing  complex 
products.  Presently,  the  nature  and  extent  of  change  propagation  is  neither  completely 
understood  nor  predictable.  Clarkson  et  al.  [2001]  shows  that  Design  Structure  Matrices 
(DSM),  known  design  method  used  to  store  information  about  connections  between 
components  and  modules  in  the  product,  can  be  used  to  provide  indication  as  how 
change  may  propagate  through  a  product.  The  proposed  mathematical  model  by 
Clarkson  et  al.  can  be  used  to  predict  the  risk  of  change  propagation  in  terms  of 
likelihood  and  impact  of  change. 

A  similar  model  to  determine  change  propagation  is  developed  by  Arts  et  al.  [2008]. 
They  formulate  a  method  to  cluster  components  of  an  adaptable  system  based  on  Design 
Structure  Matrices  (DSM).  For  each  scenario  or  action  plan  to  perform  adaptability,  the 
importance  of  component  interconnections  is  rated  in  a  separate  DSM  structure.  Based 
on  the  original  DSM  and  the  adaptability  DSM  components  can  be  grouped  together. 

Similar  research  is  found  in  the  work  by  Cervantes  et  al.  [2004]  and  Wilke[2008].  They 
both  propose  the  use  of  a  component/modular  based  approach.  Cervantes  et  al. 
develops  a  service  oriented  component  model  that  is  capable  of  autonomously  adapting 
at  runtime  due  to  the  dynamic  availability  of  the  services  provided  by  constituent 
components.  Wilkes  model  is  based  on  an  intelligent  multi-agent-system  that  enables 
the  system  to  react  on  events  autonomously. 

An  additional  approach  of  autonomous  system  change  is  developed  by  Berthelot  et  al. 
[2008].  They  propose  an  automatic  design  generation  methodology,  based  on  an 
adequation  algorithm  architecture  (AAA)  methodology.  Its  aim  is  to  consider 
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simultaneously  architecture  and  application  algorithm,  both  are  described  by  graphs,  to 
obtain  an  optimized  implementation  for  heterogeneous  architectures  based  on 
field-programmable  gate  arrays  (FPGAs).  The  purpose  of  this  methodology  is  to  ease 
and  speed  up  the  run-time  reconfiguration  (RTR)  implementation. 

Besides  the  above  listed  stream  of  research  there  is  a  second  stream  of  research  that 
focuses  on  as  to  what  extent  a  product  should  be  made  adaptable  in  order  to  keep  the 
product  profitable/usable.  An  early  example  of  a  flexible  design  approach  is  the  work  by 
Roser  et  al.  [1999].  They  propose  an  extension  of  the  robust  design  solution  by  adding 
flexibility  to  the  design  process.  The  reason  behind  the  extension  of  a  robust  design  is 
related  to  the  fact  the  robustness  only  deals  with  random  noise.  The  flexible  design 
approach  differs  from  the  robust  design  approach  by  trying  to  achieve  greater 
performance  and  improved  utility,  accepting  the  risk  of  design  changes.  The 
methodology  applied  in  the  model  by  Roser  et  al  uses  a  conservative  error  proof  design, 
which  can  be  created  by  combining  the  noise  and  the  error  distribution,  to  evaluate  and 
optimize  the  flexibility  of  the  actual  design. 

In  later  research  such  as  the  one  by  Olewnik  et  al.  [2006]  the  goal  is  to  aid  in  the 
development  of  a  decision  support  framework  that  maximizes  corporate  utility  while 
setting  attribute  and  budget  constraints  for  the  conceptual  design  phase.  This 
framework  draws  on  concepts  from  multi-objective  optimization,  consumer  choice 
theory,  and  utility  theory.  They  suggest  that  by  monitoring  how  the  changes  in  the  levels 
of  adaptability/robustness  affect  flexibility,  it  may  be  possible  to  gain  insight  into  the 
difficulties  of  mapping  between  the  performance  and  design  spaces. 

Other  research  focuses  on  how  to  design  to  maintain  the  competitive  advantage  despite 
environmental  change.  The  work  by  Mark  [2005]  develops  a  framework  to  increase  the 
system’s  flexibility  by  adapting  the  art  of  platforming.  This  type  of  flexible  system  will 
enable  the  customer  to  exchange  any  old/obsolete  component  for  a  component  that  is 
needed  to  coop  with  an  environmental  change.  Flexibility  of  this  type  of  system  is  then 
measured  as  the  performance  increase  (output)  corresponding  to  the  required  cost  and 
time  to  realize  the  change.  Similar  work  to  Mark  is  that  of  Cormier  et  al.  [2008]. 
However,  instead  of  focusing  on  the  consumer,  in  terms  of  providing  a  flexible  product 
or  flexible  design,  Cormier  et  al.  focuses  on  the  design  flexibility.  This  design  flexibility 
will  allow  the  engineer  to  adapt  a  product  under  changing  market  conditions.  This  for 
instance  will  allow  them  to  customize  products  based  on  customer  need  and 
requirements.  They  develop  a  metric  to  assist  with  the  evaluation  of  design  options  early 
on  in  the  design  process  by  rating  the  overall  flexibility  of  the  system  using  flow  analysis 
tables. 

The  focus  of  Chens  [1999]  work  is  on  providing  flexibility  in  the  design  process  by 
looking  for  a  range  of  solutions  that  involve  information  passing  between  players. 
Rather  than  focusing  on  a  single  point  solution  in  one  disciplines  model  they  include 
multiple  designs  for  the  performance  evaluation.  Similar  to  Chens  work,  Li  et  al.  [2004] 
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focuses  on  information  sharing.  They  develop  an  Internet-enabled  system  to  support 
collaborative  and  concurrent  engineering  (CE)  design  in  order  to  share  domain 
knowledge  between  designers  and  systems.  The  approach  is  based  on  the  seamless 
integrating  of  three  functional  modules,  i.e.,  co-design,  Web-based  visualization  and 
manufacturing  analysis  for  designers  to  conduct  CE  methodology  through  invoking 
distributed  services. 

All  of  the  above  mentioned  MPTs  have  one  deficiency  in  common.  There  is  no  analytical 
approach  to  supporting  the  decision  process  of  selecting  a  particular  process  of 
implementing  flexibility.  The  work  by  Schulz  et  al.  [1999]  provides  an  attempt  to 
answers  the  question  as  to  why,  when  and  where,  what,  and  how  changeability  should 
be  incorporated  into  a  systems  architecture.  However,  their  framework  of  design 
principals  falls  short  on  the  analytical  justification  of  selecting  one  principle  over 
another. 

The  same  goal  to  focus  on  flexibility  implementation  is  found  in  the  work  by  Bartolomei 
et  al.  [2008]  .  They  identity  key  system  aspects  that  should  be  addressed  for  leveraging 
flexibility.  The  approach  to  identity  these  key  system  aspects  is  based  on  real  option 
opportunities. 

Furthermore,  from  reviewing  the  current  literature  it  becomes  apparent  that  the  actual 
implementation  of  flexibility  at  the  system  level  is  very  limited.  This  is  mostly  due  to 
physical  limitations  at  the  design  stage. 
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3  Evaluation  of  Current  Capability  to  Value 
Flexibility 


We  now  present  three  approaches  for  valuing  flexibility,  show  their  applicability  using 
domain  driven  case  studies,  and  compare  their  strengths  and  weaknesses. 


3.1  Total  Ownership  Cost  (TOC)  Approach 

In  the  DoD  context,  Total  Ownership  Cost  (TOC)  includes  the  costs  to  research,  develop, 
acquire,  own,  operate,  and  dispose  of  a  system  [Boudreau-Naegle,  2003].  The  Weapon 
System  Acquisition  Reform  Act  of  2009  [WSARA  2009]  establishes  a  DoD  Director  of 
Cost  Analysis  and  Program  Evaluation  (CAPE),  whose  cost  analysis  scope  includes  “full 
consideration  of  life-cycle  management  and  sustainability  costs  in  major  defense 
acquisition  programs  and  major  automated  information  systems.”  DoD  Instruction 
5000.02,  Enclosure  7,  establishes  DoD  acquisition  policy  that  the  Cost  Analysis 
Improvement  Group  (CAIG)  within  the  CAPE  organization  “shall  prepare  independent 
Life  Cycle  Cost  Estimates  (LCCEs)  per  section  2434  of  Title  10,  United  States  Code.  ... 
The  Milestone  Decision  Authority  shall  consider  the  LCCE  before  approving  entry  into 
the  Engineering  and  Manufacturing  Development  (EMD)  Phase  or  the  Production  and 
Deployment  Phase”  [USD(AT&L),  2008]. 

As  a  result  of  being  required  as  part  of  the  DoD  acquisition  process,  a  TOC  approach 
provides  an  acquisition  decision-relevant  way  to  determine  appropriate  levels  of 
investments  in  flexibility.  When  representative  data  is  available,  this  can  be  done  by 
determining  the  relative  costs  of  system  development,  operations,  and  support  with  and 
without  the  flexibility  investments  over  a  given  system  life  span.  The  major  sources  of 
added  ownership  cost  due  to  shortfalls  in  flexibility  are  rework  during  development, 
adaptation  to  change  during  operations  and  support,  and  duplication  of  effort  in 
developing  and  supporting  similar  systems. 


3.1 .1  Advantages,  Challenges,  and  Strategies  for  the  TOC  Approach  to 
Valuing  Flexibility 

TOC  Advantages  for  Valuing  Flexibility 

Besides  being  required  by  law  and  DoD  policy,  the  TOC  approach  has  several 
advantages  with  respect  to  alternative  methods  such  as  real  options,  insurance-based, 
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and  risk-based  approaches.  It  is  easy  to  understand,  has  clear  cause-and-effect 
relationships,  can  be  used  to  complement  and  contextualize  alternative  methods,  and 
can  be  tailored  to  particular  domains. 

It  is  easy  to  understand  across  various  acquisition  stakeholders  (program  managers, 
oversight  managers,  systems  engineers,  cost  analysts,  contract  managers,  warfighters) 
because  its  costs  (investments  in  reducing  downstream  rework,  change  processing,  and 
duplication  of  effort)  and  benefits  (the  effects  of  the  investments  on  TOC)  are  expressed 
in  simple  arithmetic  formulas  as  compared  to  complex  mathematical  formulas  and 
probabilistic  assumptions.  Its  cause-effect  relationships  (the  investments  and  their 
effects  on  reducing  downstream  rework,  change  processing,  and  duplication  of  effort) 
are  straightforward  and  can  be  calibrated  to  project  data.  It  can  be  used  either  in 
standalone  mode  or  to  complement  and  contextualize  alternative  methods  such  as  real 
options,  insurance-based,  and  risk-based  approaches.  And  once  experience  and  data 
are  available  in  particular  domains,  their  relative  investment  costs  and  ownership  costs 
can  be  more  accurately  calibrated  and  related  to  domain-specific  cost  drivers  and 
decision  alternatives. 

TOC  Challenges 

The  primary  challenges  involved  in  TOC  analysis  involve  various  “devils  in  the  details” 
in  estimating  flexibility  investment  costs  and  their  resulting  cost  savings,  and  in 
predicting  uncertain  futures. 

Flexibility  investment  costs  for  individual  systems  include  the  costs  involved  in 
designing,  developing,  and  evolving  systems  to  ensure  continuing  diagnosability, 
accessibility,  replacability,  low  logistic  costs,  code  understandability  and  modularization 
around  anticipated  sources  of  change.  These  added  costs  may  be  more  than  the  added 
costs  of  design,  development,  and  evolution,  as  there  may  be  additional  costs  involved  in 
tradeoffs  between  achieving  flexibility  and  achieving  other  desired  attributes  such  as 
performance,  dependability,  usability,  size,  weight,  and  power  consumption.  Similar 
challenges  are  involved  in  estimating  the  resulting  benefits. 

Such  tradeoffs  among  desired  levels  of  service  or  “ilities”  are  classic  challenges  in 
systems  engineering.  Once  rough  models  for  determining  such  tradeoffs  are  available, 
though,  they  can  be  calibrated  and  extended  as  measured  experience  is  built  up. 

Flexibility  investment  costs  for  product  lines  include  the  costs  of  domain  engineering  to 
characterize  the  commonalities  and  variabilities  across  the  domain;  the  costs  of 
determining  how  broad  a  scope  across  which  to  develop  and  maintain  a  product  line; 
the  costs  of  developing  more  reusable  components;  the  costs  of  verifying  that  the 
reusable  components  will  operate  satisfactorily  across  the  product  line;  the  costs  of 
operating  a  repository  of  reusable  components;  and  the  costs  of  evolving  the  product 
line  architecture  and  the  reusable  components  [Boehm-Scherlis,  1992;  Poulin,  1998]. 
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Again,  the  costs  may  include  effects  of  tradeoffs  between  product  line  generality  and 
other  desired  -ilities.  The  resulting  cost  savings  will  depend  on  the  relative  costs  of  the 
commonalities  and  the  variabilities,  which  may  not  be  constant  across  the  product  line, 
and  the  degree  to  which  the  commonalities  result  in  components  that  can  be  reused 
without  modification  (black-box)  or  which  require  some  modification  (white  or  glass 
box).  The  effects  of  evolution  of  reusable  COTS  or  purchased  services  involve  more 
effort  that  reduces  the  benefits. 

A  final  challenge  is  the  difficulty  of  predicting  the  useful  lifetime  of  reuse  architectures 
and  reusable  components  in  a  world  of  rapid  change.  Again,  though,  once  rough  models 
for  determining  such  effects  are  available,  they  can  be  calibrated  and  extended  as 
measured  experience  is  built  up. 

Strategies  to  Address  TOC  Challenges 

An  important  strategy  is  to  develop  concepts  of  operation  for  the  use  of  the  TOC  models 
in  DoD  decision  situations,  and  to  prioritize  model  capabilities  that  best  support  the 
decision  situations.  Our  initial  concept  of  operation  is  one  in  which  DoD  leadership  is 
evaluating  a  proposed  system  development  approach  as  part  of  the  guidance  in  DoDI 
5000.02,  and  wishes  to  determine  how  well  the  proposers  have  analyzed  the 
alternatives  of  building  in  single-system  flexibility  or  of  developing  a  product  line  of 
similar  systems,  in  terms  of  DoD  total  ownership  costs.  Other  concepts  of  operation 
involve  internal  decisions  of  determining  how  much  single-system  flexibility  or  product¬ 
line  breadth  is  enough;  performing  tradeoffs  among  flexibility  and  additional  -ilities; 
and  decisions  to  replace  inflexible  legacy  systems  with  more  flexible  ones. 

A  particular  key  strategy  is  to  tailor  analysis  approaches  to  common  situations.  These 
generally  involve  domain  analysis,  which  is  one  of  the  elements  of  determining  the 
system’s  or  product  line’s  domain  architecture.  Others  involve  decisions  on  whether 
and  to  what  extent  to  use  commercial-off-the-shelf  (COTS)  components,  open  source 
components,  and  government -furnished  components. 

Another  strategy  involves  the  development  and  evolution  of  parametric  estimation 
models  for  investment  costs  and  resulting  ownership  costs  as  a  function  of  investment 
effectiveness.  Once  these  are  available,  they  can  be  continually  improved  via  data 
collection  and  analysis. 

The  sections  below  provide  example  models  which  have  been  calibrated  to  DoD- 
representative  project  data  that  can  be  used  as  starters  for  such  continuous 
improvement.  The  current  data  is  software-intensive  system  data  available  at  USC.  We 
are  working  with  AFIT  and  NPS  to  obtain  counterpart  hardware  data,  initially  with  AFIT 
on  modular  munitions,  and  with  NPS  on  ShipMain  maintenance  data. 
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Initial  Analysis  Focus:  Largely  Predictable  Change;  Individual  Systems  and  Product 

Lines 


Our  initial  analysis  focus  is  on  two  simple  tools  for  valuing  flexibility  via  TOC  analysis 
for  two  primary  cases  in  which  some  of  the  sources  of  development  rework  and  of  post¬ 
development  adaptation  to  change  are  relatively  predictable.  The  first  case  is  for  an 
individual  system  in  which  common  sources  of  change  can  be  anticipated  and 
approaches  for  facilitating  the  sources  of  change  can  be  invested  in.  For  hardware¬ 
intensive  systems,  this  is  covered  under  design  for  maintainability  [Blanchard  et  al., 
1995],  including  considerations  of  diagnosability,  accessibility,  replacability,  and  logistic 
cost  tradeoffs.  For  software-intensive  systems,  diagnosability  is  also  important,  but 
other  factors  such  as  code  understandability  and  modularization  around  anticipated 
sources  of  change  are  also  critical  [Lehman-Belady,  1985;  Parnas,  1979].  The  general 
effect  of  such  strategies  is  shown  in  Figure  1,  which  shows  the  exponential  growth  in  cost 
to  make  changes  vs.  time  for  traditional  large  TRW  systems  [Boehm  1981]  vs.  the  larger 
up-front  cost  to  eliminate  risks  and  design  for  ease  of  change,  and  subsequent  flat  cost- 
growth  data  for  the  later  TRW  CCPDS-R  project  [Royce,  1998]. 


The  second  case  is  for  reuse-driven  investments  across  a  family  of  systems  vs. 
development  of  individual  stovepipe  systems.  Here,  the  strategy  is  basically  the  same 
for  hardware-intensive  and  software-intensive  systems.  For  a  class  of  similar  products, 
a  systems  engineering  activity  identifies  the  commonalities  among  the  products,  and 
develops  these  into  reusable  infrastructure  and  components.  It  also  identifies  the 
variabilities  among  the  products,  and  develops  plug-compatible  interfaces  from  the 
common  parts  to  the  variable  parts  to  facilitate  integration.  The  general  effect  of  such 
strategies  is  to  increase  the  cost  and  schedule  of  the  initial  projects  to  include 
investment  in  identifying  and  architecting  the  common  infrastructure  and  generalizing 
the  common  parts  for  product  line  use,  but  to  significantly  reduce  the  costs  and 
schedules  of  later  products  in  the  product  line  through  higher  reuse  and  simpler 
integration.  Figure  2  shows  an  example  from  Hewlett-Packard’s  investment  in  software 
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reuse  across  its  measurement-box  and  network-equipment  product  lines  [Griss  1993; 
Lim  1998]. 

This  Section  will  elaborate  on  the  overall  advantages  and  challenges  of  the  TOC 
approach  with  respect  to  valuing  flexibility  in  Section  3.1.1,  and  will  then  describe 
working  models  and  their  use  in  analyzing  the  TOC  and  return-on-investment  (ROI) 
effects  of  the  individual-system  approach  in  Section  3.1.2,  and  of  the  product  line 
approach  in  Section  3.1.3. 


Month  and  Year  Released 


Figure  2:  HP  Product  Line  Reuse  Investment  and  Payoff 


3.1 .2  A  TOC  Model  for  Valuing  Flexibility  of  Individual  Systems 


Reducing  Software  Rework  via  Architecting  for  Flexibility 

Analysis  of  project  defect  tracking  cost-to-fix  data  (a  major  source  of  rework  costs)  on 
representative  traditional  TRW  C4lSR-type  projects  showed  a  Pareto  effect,  as  20%  of 
the  defects  accounted  for  80%  of  the  rework  costs,  and  that  these  20%  were  primarily 
due  to  inadequate  architecting  for  flexibility  and  risk  resolution. 
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For  example,  in  TRW  Project  A  in  Figure  3,  most  of  the  rework  was  the  result  of 
development  of  the  network  operating  system  to  a  nominal-case  architecture,  and 
finding  that  the  systems  engineering  of  the  architecture  neglected  to  address  the  risk 
that  the  operating  system  architecture  would  not  support  the  project  requirements  of 
successful  system  fail-over  if  one  or  more  of  the  processors  in  the  network  failed  to 
function.  Once  this  was  discovered  during  system  test,  it  turned  out  to  be  an 
“architecture-breaker”  causing  several  sources  of  expensive  rework  to  the  already- 
developed  software.  A  similar  “architecture-breaker,”  the  requirement  to  handle  extra- 
long  messages  (e.g.,  full-motion  video),  was  the  cause  of  most  of  the  rework  in  Project  B, 
whose  original  nominal-case  architecture  assumed  that  almost  all  messages  would  be 
short  and  easy  to  handle  with  a  fully  packet-switched  network  architecture. 


(/) 

k 

Q_ 

V) 

X 


C/5 

O 

o 

<4- 

o 

nP 


0  10  20  30  40  50  60  70  80  90  100 


%  of  Software  Problem  Reports  (SPR’s) 

Figure  3:  Pareto  80-20  Distribution  of  Cost-to-Change 


Earlier,  analyses  of  cost-to-fix  data  at  IBM[Fagan  1976],  GTE  [Daly  1977],  Bell  Labs 
[Stephenson  1976],  and  TRW  [Boehm  1976]  found  consistent  results  showing  the  high 
payoff  of  finding  and  fixing  defects  as  early  as  possible.  As  seen  in  Figure  4,  relative  to 
an  effort  of  10  units  to  fix  a  requirements  defect  in  the  Code  phase,  fixing  it  in  the 
Requirements  phase  involved  only  about  2  units  of  effort,  while  fixing  it  in  the 
Operations  phase  involved  about  100  units  of  effort,  sometimes  going  as  high  as  800 
units.  These  results  caused  TRW  to  develop  policies  requiring  thorough  risk  analyses  of 
all  requirements  by  the  project’s  Preliminary  Design  Review  (PDR),  including  such  cost- 
to-change  risks  as  off-nominal  architecture-breakers  and  user  interfaces.  With  TRW’s 
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adoption  of  the  Ada  programming  language  and  associated  ability  to  verify  the 
consistency  of  Ada  module  specifications,  the  risk  policy  was  extended  into  an  Ada 
Process  Model  for  software,  also  requiring  that  the  software  architecture  pass  an  Ada 
compiler  module  consistency  check  prior  to  PDR  [Royce,  1998],  enabling  much  of 
systems  integration  to  be  done  before  component  development. 


Phase  in  which  defect  was  fixed 

Figure  4:  Cost  of  Change  vs.  Development  Phase,  Traditional  Sequential  Development 


A  Successful  Example:  CCPDS-R 

This  enabled  subsequent  projects  to  perform  much  of  systems  integration  before 
providing  the  module  specifications  to  programmers  for  coding  and  unit  test.  As  a 
result  of  this  and  the  elimination  of  architecture  risks  prior  to  PDR,  subsequent  projects 
were  able  to  significantly  reduce  late  architecture-breaker  rework  and  the  steep  slope  of 
the  cost-to-fix  curve.  A  good  example  was  the  Command  Center  Processing  and  Display 
System- Replacement  (CCPDS-R)  project  described  in  [Royce,  1998]  whose  flattened 
cost-to-fix  curve  was  shown  in  Figure  1.  It  delivered  over  a  million  lines  of  Ada  code 
within  its  original  budget  and  schedule.  Its  PDR  was  held  in  month  14  of  a  35-month 
initial-delivery  schedule  and  included  about  25%  of  the  initial-delivery  budget,  including 
development  and  validation  of  its  working  high-risk  software,  such  as  its  network 
operating  system  and  the  key  portions  of  its  user  interface  software. 

Contract  Number:  H98230-08-D-01 71  DO  02,  TO  02  RT018 

Report  No.  SERC-2010-TR-010 
09/25/2010 


UNCLASSIFIED 

35 


UNCLASSIFIED 


Data  from  Projects  A,  B,  and  CCPDS-R  have  been  used  to  develop  and  calibrate  a  model 
that  compares  the  TOC  of  projects  like  A  and  B  which  have  not  been  designed  for 
flexibility,  and  CCPDS-R,  which  was  designed  for  flexibility.  The  model  and  examples  of 
its  use  are  shown  below. 

The  Total  Ownership  Cost  -  Single  SvstemfTOC-SS)  Model 

The  simple  initial  TOC-SS  model  has  the  following  inputs: 

%D  -  The  %  of  development  cost  invested  in  Design  for  Flexibility 

System  Size  -  For  software,  the  equivalent  KSLOC  (thousands  of  source  lines  of 
code) 


-  For  hardware,  the  COSYSMO  size  parameter:  complexity-weighted 
numbers  of 

-  requirements,  interfaces,  operational  scenarios,  and  algorithms 
#F  -  The  number  of  years  that  the  system  undergoes  field  changes 
%FC  -  The  percentage  of  the  fielded  system  size  undergoing  change 

The  TOC-SS  model  has  the  following  outputs: 

TOC  (Devel)  -  The  TOC  for  development 

TOC  (Devel  +  K)  -  TOC  (Devel)  +  TOC  (K  years  of  fielding),  K  =  l, ...,  #F 
The  steps  in  the  simple  initial  TOC-SS  model  are  as  follows: 

l.  For  design  with  and  without  flexibility,  use  %D  as  a  proxy  for  Design  for 
Flexibility  to  determine  the  exponent  B  determining  the  project’s  Rework 
Fraction  RF  from  its  size,  using  the  values  in  the  table  below.  For  software,  B 
has  been  calibrated  to  161  projects  in  COCOMO  II  [Boehm  et  al.,  2000].  For 
hardware,  we  propose  to  calibrate  it  from  AFIT  or  NPS  data. 


%D 

5 

10 

17 

25 

33 

50 

B 

.0707 

•0565 

.0424 

.0283 

.0141 

0.0 

2.  Compute  RF  =  (Size1+B  -  Size)  /  Size 
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3.  Estimate  the  project’s  nominal  cost  NC  from  its  Size  and  other  parameters.  For 
software,  COCOMO  II  or  other  cost  models  can  be  used.  For  hardware,  cost 
models  such  as  Price  H  and  SEER-H  can  be  used. 

4.  Compute  TOC  (Devel)with  and  without  flexibility  from  their  values  of  RF  and 
%D  as  TOC  (Devel)  =  %D  +  NC  *  (1  +  RF) 

5.  For  K  =  1, ...,  #F,  compute  TOC  (Devel  +  K)  =  TOC  (Devel)  +  K*  NC*  RF 

Comparison  of  Model  Results  and  Projects  A.  B,  and  CCPDS-R  Data 

Figures  5  and  6  below  show  the  results  of  calculating  the  relative  Total  Ownership  Costs 
for  Systems  A,  B,  and  C  (CCPDS-R).  For  comparison,  the  values  of  %Rework  (RF*ioo) 
for  Systems  A  and  B  are  ioo107°7-  100  =  38.5%  (vs  35.7%  and  41,2%),  and  for  CCPDS-R 
is  (35510283-355)/3-55  =  18%  (vs.  13.85%). 


A 

B 

C 

D 

E 

1 

Input  Parameters 

System 

2 

A 

B 

c 

3 

Software  Size  (KSLOC) 

100 

100 

355 

4 

#  Change  Requests/Release 

373 

1005 

1600 

5 

# Change  Requests  (l&T only) 

6 

S  l&T  Change  Requests/Release/  >  1  PM 

27 

22 

7 

#Total  Change  Requests/Release/  >1  PM 

16 

8 

Change  Request  Fix  Time  (See  assumption  #2) 

261 

356 

263 

9 

Total  Effort  (Person  Months) 

731 

865 

1900 

10 

%  Arch,  RESL 

5% 

5% 

25% 

11 

%  Rework,  RVOL 

35.70% 

41.16% 

13.85% 

12 

13 

Cumulative  Total  Cost  of  Ownership 

Project  A 

Project  B 

Project  C 

14 

Cycle  1 

40.70% 

46.16% 

38.85% 

15 

Cycle  2 

76.41% 

87.31% 

52.70% 

16 

Cycle  3 

112.11% 

128.47% 

66.55% 

17 

Cycle  4 

147.82% 

169.62% 

80.40% 

18 

Cycle  5 

183.52% 

210.78% 

94.25% 

Figure  5:  TOC  Calculations  for  Projects  A,  B,  and  C  (CCPDS-R) 
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Figure  6  shows  the  results  graphically.  Thus,  the  model  can  be  used  in  an  acquisition 
decision  situation  to  show  that  if  a  project  proposes  a  hastily-designed,  inflexible  point 
solution  and  has  not  done  an  analysis  of  the  alternative  of  investing  in  determining  its 
primary  sources  of  change  and  designing  to  confine  these  within  system  components, 
the  project’s  TOC  will  represent  a  significantly  higher  cost  to  DoD  and  the  taxpayers. 


- Project  A  - Project  B  - Project  C 


Figure  6:  TOC’s  for  Projects  A,  B,  and  C  (CCPDS-R)  Relative  to  Baseline  Costs 


3.1 .3  The  TOC  Model  for  Valuing  Flexibility  of  Product  Lines 

This  approach  is  a  TOC  analysis  for  a  family  of  systems.  The  value  of  investing  in 
product-line  flexibility  using  Return-On-Investment  (ROI)  and  TOC  is  assessed  with 
parametric  models  adapted  from  the  Constructive  Product  Line  Investment  Model 
(COPLIMO)  [Boehm  et  al.,  2004].  COPLIMO  is  based  on  the  well-calibrated  COCOMO 
II  model  [Boehm  et  al.,  2000]  with  161  data  points.  The  new  models  are  implemented 
in  separate  tools: 
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•  System-level  product  line  flexibility  investment  model. 

•  Software  product  line  flexibility  investment  model.  The  detailed  software  model 
includes  schedule  time  with  NPV  calculations. 


Figure  7  shows  the  inputs  and  outputs  for  the  system-level  product  line  model. 


For  Set  of  Products: 

•  Average  Product  Cost 

•  Annual  Change  Cost 

•  Ownership  Time 

•  Percent  Mission- 
Unique,  Adapted, 
Reused 

•  Relative  Cost  of 
Developing  for  PL 
Flexibility  via  Reuse 

•  Relative  Costs  of 
Reuse 


Systems 


PL  Flexibility 
Value  Model  ’ 


1  As  Functions  of  # 

I  Products,  it  Years  in 
I  Life  Cycle: 

PL  Total  Ownership 
Costs 

PL  Flexibility 
Investment 
PL  Savings  (ROI) 


Figure  7:  Systems  product  line  flexibility  value  model  (TOC-PL). 


The  cost  of  the  first  system  is  determined  by  multiplying  the  average  product  cost  by  the 
fraction  of  the  product  to  be  developed  for  reuse,  (%Adapted  +  %Reused)/ioo, 
multiplying  that  by  the  relative  cost  of  developing  for  product  line  flexibility  reuse,  and 
adding  that  to  the  system-unique  cost  (%Unique  *  Average  Product  Cost  /  100)  which 
does  not  have  to  be  developed  for  reuse.  For  subsequent  products,  the  cost  of  the 
unique  system  portion  is  the  same,  but  the  equivalent  costs  of  adapted  and  reused 
portions  are  determined  by  their  relative  costs  of  reuse.  For  hardware,  the  relative  costs 
of  reuse  should  include  not  only  the  cost  of  adapting  the  reused  components,  but  also 
the  carrying  costs  of  the  inventory  of  reusable  components  kept  in  stock. 


The  net  effort  savings  for  the  product  line  are  the  cost  of  developing  separate  products 
(#Products*Average  Product  Cost)  minus  the  total  cost  of  developing  Product  1  for  reuse 
plus  developing  the  rest  of  the  products  with  reuse.  The  ROI  for  a  system  family  is  the 
net  effort  savings  divided  by  the  product  line  flexibility  investment,  (Average  Product 
Cost)  *  (%Adapted  +  %Reused)  *  (Relative  Cost  of  Reuse  +  Carrying  Cost  Fraction  - 
i)/ioo.  The  TOC  is  computed  for  the  total  lifespan  of  the  systems  and  normalized  to  net 
present  value  at  specified  interest  rates. 


The  tools  are  available  for  SERC  use  with  a  file  system,  and  are  awaiting  clearance  for 
public  distribution. 
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3.1 .4  Quantitative  Evaluation  of  Flexibility  Using  a  Case  Study 

The  case  shown  below  represents  a  family  of  seven  related  systems  with  three-year 
ownership  durations.  It  is  assumed  annual  changes  are  10%  of  the  development  cost. 
Within  the  family  of  systems,  each  is  comprised  of  40%  unique  functionality,  30% 
adapted  from  the  product  line  and  30%  reused  as-is  without  changes.  Their  relative 
costs  are  40%  for  adapted  functionality  and  5%  for  reused.  The  up-front  investment  cost 
in  flexibility  of  1.7  represents  70%  additional  effort  compared  to  not  developing  for 
flexibility  across  multiple  systems.  Figure  8  shows  the  consolidated  TOC  and  ROI 
outputs. 


Systems  Product  Line  Flexibility  Value  Model 

*•>%!(  >t  •  I..V.il*Vl  I  ItKNI 

Welcome  SERC  Collaborator 


Preferences 


Open  '  Save  Save  As 


System  Costs 

Average  Product  Development  Cost  (Burdened  SMI  j  5  Ownership  Time  (Years)  3 

Annual  Change  Cost  (%  of  Development  Cost)  10  Interest  Rate  (Annual  %)  7 

Product  Une  Percentages  Relative  Costs  of  Reuse  (%) 

Unique  %  40  Relative  Cost  of  Reuse  for  Adapted  40 

Adapted  %i  30  Relative  Cost  of  Reuse  for  Reused  5 

Reused  %  30 


Investment  Cost 


Relative  Cost  of  Developing  for  PL  Flexibility  via  Reuse  1.7  Sensitivity  '  otf  ;| 


<  Calculate  I 


Results 


#  of  Products 

1 

2 

3 

4 

5 

6 

7 

Development  Cost  (SM) 

S7.1 

$2  7 

$2.7 

$2  7 

$2.7 

$2  7 

$2.7 

Ownership  Cost  ($M) 

S2.1 

$0.8 

$0  8 

$0.8 

$0.8 

$0.8 

$0.8 

Cum.  PL  Cost  ($M) 

S9.2 

$12.7 

$16.2 

$19.7 

$23.1 

$26.6 

$30.1 

PL  Flexibility  Investment  ($M) 

$2.1 

$0 

$0 

$0 

$0 

$0 

$0 

PL  Effort  Savings 

(*2  7) 

$0  3 

$3.3 

$6  3 

$9  4 

$12.4 

$15.4 

Return  on  Investment 

-1  30 

0  14 

1  58 

302 

4  46 

5  90 

7  34 

Return  on  Investment 


| -1.3 1|  0.1  irrell  3.Q|j*5|[  5.9 1 1  7.3 1 

1  2  3  4  5  6  7 


Product# 

Figure  8:  Product  line  flexibility  TOC  and  ROI  results. 
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However,  it  is  desired  to  evaluate  ranges  of  options  and  assess  the  sensitivity  of  TOC. 
The  tools  allow  for  a  range  of  relative  costs  as  shown  in  Figure  9  for  sensitivity  runs. 
The  results  show  that  the  model  can  help  projects  determine  “how  much  product  line 
investment  is  enough”  for  their  particular  situation.  In  the  Figure  9  situation,  the  best 
level  of  investment  in  developing  for  reuse  is  an  added  60%. 


Investment  Cost 

Relative  Cost  of  Developing  for  PL  Flexibility  via  Reuse  1  Min  Max  H  Runs  Sensitivity  On  T] 

Calculate  ' 


ROI  Sensitivity  Results 


1.2  1.4  1.6  1.8  2.0 

Investment 

Figure  9:  Example  sensitivity  analysis  (ROI  only). 

Other  types  of  sensitivity  analyses  can  be  conducted.  Figure  10  shows  example  results  of 
assessing  the  sensitivity  of  TOC  across  a  range  of  product  ownership  durations.  Most 
product  line  cost  estimation  models  focus  just  on  development  savings,  and 
underestimate  the  savings  in  Total  Ownership  Costs. 


Figure  10:  TOC-PL  sensitivity  by  ownership  duration  results. 


Contract  Number:  H98230-08-D-01 71 

Report  No.  SERC-2010-TR-010 
09/25/2010 


DO  02,  TO  02  RT  018 


UNCLASSIFIED 

41 


UNCLASSIFIED 


The  TOC-PL  model  can  also  be  used  in  an  acquisition  decision  situation  to  show  that  if  a 
project  proposes  a  stovepipe  single-product  point  solution  in  an  area  having  numerous 
similar  products,  and  has  not  done  an  analysis  of  the  alternative  of  investing  in  a 
product  line  approach,  the  project’s  TOC  will  represent  a  significantly  higher  cost  to 
DoD  and  the  taxpayers. 


3.1 .5  Proposed  Next  Steps 

Even  in  their  current  simple  forms,  the  TOC-SS  and  TOC-PL  models  provide  strong 
capabilities  for  analyzing  alternative  approaches  to  system  acquisition  and  the  effects  on 
TOC.  We  plan  to  work  with  AFIT  and  NPS  personnel  and  data  such  as  Modular 
Munitions  and  ShipMain  data  to  calibrate  and  refine  the  models  to  better  address 
hardware-intensive  systems.  Depending  on  sponsor  priorities,  there  are  several  further 
extensions  to  the  models  that  could  add  value  in  various  situations.  The  TOC-SS  model 
could  address  situations  in  which  annual  change  traffic  varies  by  year  or  by  system 
element,  provide  present  value  calculations  (done  in  TOC-PL),  provide  special  domain- 
specific  models,  or  investigate  complementary  strategies  such  as  investigating  key 
personnel  strategies,  use  of  COTS  products  or  purchased  services.  Similar  extensions 
could  be  added  to  the  TOC-PL  model,  including  effects  of  varying  product  sizes,  change 
rates,  product  line  investment  costs,  and  degrees  of  reuse  across  the  products  in  the 
product  line.  The  models  could  be  combined  with  each  other  or  with  complementary 
models  involving  real  options,  risk  assessments,  or  tradeoffs  among  flexibility  aspects 
such  as  evolvability,  interoperability,  portability,  or  reconfigurability;  or  between 
flexibility  aspects  and  other  -ilities  such  as  security,  safety,  performance,  reliability,  and 
availability. 

More  ambitious  extensions  could  address  aspects  of  complex  adaptive  systems  to 
address  unforeseeable  sources  of  change,  such  as  monitoring  change  traffic  to  determine 
changes  or  patterns  in  system  hot-spots  and  ways  to  adapt  the  system  to  better  deal  with 
them,  or  to  address  complementary  adaptive  approaches  such  as  autonomous  logistics 
or  monitoring  of  threat  patterns. 


3.1 .6  Strengths  and  Weaknesses 


Strengths 

•  Simple  approach 

•  Including  future  uncertainty  in  decision 

•  Cooperating  with  the  neoclassical  theory  of  investment,  which  states  when  the 
marginal  cost  and  marginal  revenue  are  identical,  the  efficiency  of  economics  in 
achieved. 
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Weaknesses 

•  Estimating  future  cash  flow  is  problematic. 

•  How  the  discount  rate  is  determined  is  controversial. 

•  The  attempt  to  overcome  these  2  disadvantages  is  TCO  or  expanded  NPV 

•  NPV  approach  implicitly  assumes  reversible  decision.  Reversible  investment 
means  the  investment  can  be  undone  and  the  expenditures  recovered.  However 
in  the  real  world  many  decisions  are  irreversible. 

•  NPV  approach  is  appropriate  for  ‘now  or  never’  decision. 


3.2  Hedge  Framework 


3.2.1  Overview  of  the  Approach 

The  University  of  Virginia  (UVa)  team  is  developing  and  evaluating  novel  approaches  to 
creating,  modeling,  valuing  and  exploiting  flexibility  in  system  design  and  development. 

The  guiding  assumption  is  that  the  goal  of  the  decision  maker  is  to  obtain  the  greatest 
value  possible  for  one's  investment  in  a  system.  The  question  is,  where  does  value  come 
from,  and  how  can  one  reason  systematically  to  optimize  for  value? 

Value  in  UVa's  view  is  measured  by  a  rational  and  well  informed  actor's  willingness  to 
pay  for  an  asset  in  a  given  environment.  The  role  of  the  environment  in  this  model  is 
crucial.  UVa's  framework  thus  emphasizes  the  need  to  model  the  assumed  environment 
when  reasoning  about  value. 

The  next  question  is,  what  is  the  value  of  an  engineering  system  and  where  does  it  come 
from?  In  UVa's  framework,  the  value  of  a  system  derives  from  two  basic  sources:  the  set 
of  capabilities  that  the  system  provides  and  the  set  of  opportunities  that  it  provides. 

A  capability  describes  what  a  system  can  do  as  is.  An  opportunity  is  the  possibility  to 
invest  additional  resources  to  change  a  system  into  a  new  system  -  as  it  could  be.  To 
exercise  an  opportunity— to  actually  make  such  a  change— incurs  some  cost  and  yields  a 
system  with  a  modified  capability  and/or  opportunity  set.  The  changes  in  capabilities 
and/or  opportunities  generally  then  change  the  value  of  the  system. 

We  can  now  state  two  key  premises  of  this  work.  First,  the  opportunity  set  that  a  system 
provides— the  possibilities  a  system  affords  for  follow-on  investments  in  new  capabilities 
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or  improved  performance  or  in  other  improvements— can  have  enormous  value.  Indeed, 
its  value  can  far  exceed  that  of  its  capabilities,  even  in  cases  in  which  no  opportunity  can 
be  taken  for  immediate  or  certain  profit.  Second,  it  is  not  enough  to  have  opportunities. 
One  must  also  exercise  effectively  the  opportunities  that  one  has  manner  to  realize  their 
potential  value. 

These  observations  then  lead  to  two  specific  questions  for  which  we  seek  constructive 
answers.  First,  what  important  set  of  opportunities  (i.e.,  what  flexibility)  does  a  system 
provide  and  what  is  it  worth  in  a  given  environment?  Second,  given  an  opportunity  set 
in  a  changing  environment,  when,  if  ever  does  one  decide  to  exploit  a  given  opportunity? 

The  problem  that  UVa  is  addressing  is  that,  while  it  is  relatively  easy  for  decision  makers 
see  and  value  capabilities,  it  is  difficult  to  see  and  value  opportunities.  Capabilities  are 
useful,  visible  properties  of  a  system  as  it  is.  Opportunities,  by  contrast,  are  invisible— 
often  rooted  in  internal  details  of  a  system  or  project,  e.g.,  in  a  modular  architecture— 
and  often  have  no  immediate  relevance  to  system  users.  Consequently,  the  opportunity 
set  provided  by  a  systems  is  often  overlooked,  its  value  is  underestimated,  the  system  is 
valued  incorrectly,  and  managerial  decisions  deliver  much  less  value  than  possible. 

UVa's  approach  to  this  problem  is  to  make  both  opportunity  sets  and  the  environments 
in  which  they  will  be  valued  explicit  in  system  design  and  engineering.  An  environment 
in  turn  models  not  only  the  present  “state  of  nature”  but  also  possible  future  states. 

It  is  key  to  value  opportunities  with  respect  to  environments  that  include  projections 
about  possible  future  states  of  nature.  First,  the  present  value  of  an  opportunity  utterly 
depends  on  projections  about  possible  future  conditions.  Second,  we  must  have  a  good 
handle  on  the  present  value  of  an  opportunity  in  order  to  develop  a  valid  theory  about 
when,  if  ever,  to  exercise  it.  The  decision  rule  in  this  regard  is  that  one  should  exercise  if 
and  only  if  the  payoff  net  of  investment  costs  equals  or  exceeds  the  present  value  of  the 
opportunity. 

UVa's  technical  approach  is  to  make  key  opportunities  and  environment  models  explicit 
during  system  design  and  evolution,  and  thereby  subject  to  both  valuation  and  dynamic 
management.  The  valuation  of  the  opportunity  sets  afforded  by  a  system  is  our  answer 
to  the  question  what  is  flexibility  and  how  should  it  be  valued?  Dynamic  updating  of  an 
environment  model  and  enforcement  of  the  aforementioned  rule  is  UVa's  answer  to  the 
question,  given  such  flexibility,  how  does  one  use  it  to  optimize  project  value  over  time? 

Concretely,  UVa  is  developing  these  ideas  primarily  in  the  context  of  software  systems. 
In  this  context,  UVa  is  exploring  enhancements  to  modern  software  development  tools 
and  environments  to  provide  support  for  explicit  representation  of  sets  of  opportunities, 
environments,  and  valuation  functions.  In  terms  of  valuation  functions,  UVa  is  working 
on  the  assumption  that  there  is  considerable  uncertainty  about  what  specific  valuation 
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approaches  are  best.  Therefore,  UVa  remains  somewhat  neutral,  and  is  exploring  an 
approach  that  will  accommodate  different  approaches  as  modular  plug-ins. 

To  ground  the  work,  UVa  is  exploring  the  use  of  event  trees  with  subjective  probabilities 
to  value  opportunities  created  by  decisions  to  delay  investments  while  uncertainties  are 
resolved;  and  the  use  of  Baldwin  and  Clark's  approach  to  valuation  of  the  opportunities 
for  low-cost  substitution  created  by  modular  architectures. 

UVa  is  well  aware  of  theoretical  difficulties  in  assuming  that  traditional  “real  options” 
techniques  can  be  applied  to  engineering  design  decisions  -  approaches  that  assume 
replicating  portfolios  or  proxies  for  such,  or  that  assume  specific  characteristics  of  the 
underlying  stochastic  payoff  processes.  The  specific  valuation  approaches  that  we  are 
exploring  are  explicitly  intended  to  avoid  such  difficulties.  Indeed,  our  use  of  the  term 
opportunity  as  opposed  to  option  is  meant  to  clearly  indicate  that  we  do  not  propose  to 
adopt  a  real  options  approach  uncritically. 

In  sum,  UVa  views  flexibility  in  terms  of  opportunities  to  make  investments  to  change  a 
system  in  some  way.  The  value  of  these  opportunities,  and  thus  the  value  of  flexibility,  is 
highly  sensitive  to  assumptions  about  the  environment  (e.g.,  willingness  of  customers  to 
pay  for  particular  capabilities  at  particular  times),  including  assumptions  about  possible 
future  states  of  the  environment.  The  value  of  flexibility  is  the  value  of  the  opportunities 
in  a  design.  Beyond  assessing  such  value,  decision  makers  need  to  have  the  analysis  and 
the  capabilities  to  exploit  opportunities  effectively  to  realize  their  latent  value.  To  help 
engineers  and  decision  makers  to  reason  in  these  terms,  UVa  is  developing  an  approach 
and  supporting  tools  to  make  opportunities,  environments,  and  valuation  techniques 
explicit  and  subject  to  scientific,  value-driven  management  during  system  development. 


3.2.2  Quantitative  Evaluation  of  Flexibility  Using  a  Case  Study 

UVa  is  still  early  in  the  development  of  its  approach  and  supporting  tools,  and  has  not 
yet  conducted  in-depth  case  studies. 


3.2.3  Strengths  and  Weaknesses 

Strengths 


•  Emphasizes  needs  to  convert  theoretical  work  into  science-based,  practical  tools 
for  working  engineers  and  decision-makers. 

•  Strongly  recognizes  that  naive  application  of  real  options  theory  is  problematical, 
while  retaining  an  overall  perspective  in  terms  of  investment  under  uncertainty. 

Contract  Number:  H98230-08-D-01 71  DO  02,  TO  02  RT018 

Report  No.  SERC-2010-TR-010 
09/25/2010 

UNCLASSIFIED 

45 


UNCLASSIFIED 


•  Recognizing  that  there  is  still  uncertainty  about  what  valuation  approaches  will 
be  best,  creates  opportunities  to  choose  later  through  provision  of  a  plug-in,  i.e., 
modular,  architecture. 

•  Intent  is  to  provide  several  valuation  techniques  at  the  same  time,  including  ones 
for  valuing  opportunities  created  by  decision  to  delay  investments,  and  by 
modular  design  architectures. 

Weaknesses 

•  The  usability  of  event  trees  with  subjective  probability  estimates  for  modeling 
environments  including  uncertain  future  states  of  nature  remains  unclear. 

•  Continual  updating  of  environment  models  as  uncertain  conditions  resolve  to 
certain  outcomes  could  be  burdensome. 

•  The  provision  of  valid  subjective  estimates  is  difficult  and  subject  to  gaming. 

•  There  are  innumerable  planned  and  unplanned  opportunities  in  any  project; 
picking  the  ones  to  model  will  pose  challenges. 

•  Non-software  engineers  will  not  have  ready  access  to  the  tools  UVa  is  developing. 
UVa  is  exploring  the  use  of  Web  2.0  interfaces  to  supplement  the  programming 
interfaces  that  will  be  provided  by  UVa's  “Eclipse”-based  prototypes. 

•  Most  real  options  and  related  approaches  assume  risk-neutral  decision  makers 
implicitly  or  explicitly.  This  assumption  need  to  be  checked  in  a  risk  management 
sense. 


3.3  Knowledge  Value  Added  +  Integrated  Risk 
Management  and  Real  Options 


3.3.1  Overview  of  the  Approach 

KVA+IRM  analysis  is  designed  to  support  technology  portfolio  acquisitions  and  to 
empower  decision-makers  by  providing  performance-based  data  and  scenario  analysis. 
With  historical  data  provided  by  KVA,  potential  strategic  investments  can  then  be 
evaluated  with  Integrated  Risk  Management  Analysis. 

The  KVA+IRM  valuation  framework  measures  operating  performance,  cost- 
effectiveness,  return  on  investments,  risk,  real  options  (capturing  strategic  flexibility), 
and  analytical  portfolio  optimization.  The  framework’s  components  of  data  collection, 
KVA  methodology,  and  Integrated  Risk  Management  analysis  collectively  provide 
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performance-based  data  and  analyses  on  individual  projects,  programs  and  processes 
within  a  portfolio  of  IT  investments. 


KVA  METHODOLOGY 

REAL  OPTIONS  THEORY 

Step  1 :  Calculate  Time  to  Learn. 

Step  2:  Calculate  Value  of  Output  (K)  for  each  sub¬ 
process. 

Step  3:  Calculate  Total  K  for  process. 

+ 

Stepl:  Risk  Identification 

List  of  projects  and  strategies  to  evaluate. 

Step  2:  Risk  Prediction 

Base  case  projections  for  each  project. 

Step  3:  Risk  Modeling 

Step  4:  Derive  Proxy  Revenue  Stream. 

Develop  static  financial  models  with  KVA  data. 

Step  5:  Develop  the  Value  Equation  Numerator  by 

assigning  revenue  streams  to  sub-processes. 

Step  4:  Risk  Analysis 

Dynamic  Monte  Carlo  simulation. 

Step  6:  Develop  value  equation  denominator  by 
assigning  costs  to  sub-processes. 

Step  5:  Risk  Mitigation 

Framing  real  options. 

Step  7:  Calculate  metrics: 

Return  on  Investment  (ROI) 

Return  on  Knowledge  Assets  (ROK) 

Step  6:  Risk  Hedging 

Options  analytics,  simulation  &  optimization. 

Step  7:  Risk  Diversification 

Portfolio  optimization  and  asset  allocation. 

Step  8:  Risk  Management 

Reports  presentation  and  update  analysis. 

Figure  11:  KVA+IRM  NPS  Valuation  Framework 


The  KVA+RO  valuation  framework  measures  operating  performance,  cost-effectiveness 
and  return  on  investments.  Its  methodology  can  be  summarized  in  the  following  table 
including  the  general  data  collection  process. 
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DATA  COLLECTION 

KVA  METHODOLOGY 

REAL  OPTIONS  THEORY 

•  Collect  baseline 
data 

Stepl:  Calculate  Time  to  Learn. 

Step  2:  Calculate  Value  of  Output  (K)  for  each  sub- 

Stepl:  Risk  Identification 

List  of  projects  and  strategies  to  evaluate. 

•  Identify  sub¬ 
processes 

process. 

Step  3:  Calculate  Total  K  for  process. 

Step  2:  Risk  Prediction 

Base  case  projections  for  each  project. 

•  Research  market 
comparable  data 

•  Conduct  market 
analysis 

Step  4:  Derive  Proxy  Revenue  Stream. 

Step  5:  Develop  the  Value  Equation  Numerator  by 

assigning  revenue  streams  to  sub-processes. 

Step  6:  Develop  value  equation  denominator  by 
assigning  costs  to  sub-processes. 

Step  3:  Risk  Modeling 

Develop  static  financial  models  with  KVA  data. 

Step  4:  Risk  Analysis 

Dynamic  Monte  Carlo  simulation. 

•  Determine  key 
metrics 

Step  7,  8,  9:  Calculate  metrics: 

Return  on  Investment  (ROI) 

Return  on  Knowledge  Assets  (ROKA) 

Step  5:  Risk  Mitigation 

Framing  real  options. 

Return  on  Knowledge  Investment  (RKOI) 

Step  6:  Risk  Hedging 

Options  analytics,  simulation  &  optimization. 

Step  7:  Risk  Diversification 

Portfolio  optimization  and  asset  allocation. 

Step  8:  Risk  Management 

Reports  presentation  and  update  analysis. 

Figure  12:  KVA+RO  NPS  Valuation  Framework 


Knowledge  Value  Added  Methodology 

A  new  paradigm  in  sub-corporate  performance  analytics,  KVA  measures  the  value 
provided  by  human  capital  assets  and  IT  assets  by  analyzing  an  organization,  process  or 
function  at  the  process-level.  It  provides  insights  into  each  dollar  of  IT  investment  by 
monetizing  the  outputs  of  all  assets,  including  intangible  knowledge  assets.  By 
capturing  the  value  of  knowledge  embedded  in  an  organization’s  core  processes, 
employees  and  IT,  KVA  identifies  the  actual  cost  and  revenue  of  a  product  or  service. 
Because  KVA  identifies  every  process  required  to  produce  an  output  and  the  historical 
costs  of  those  processes,  unit  costs  and  unit  prices  of  products  and  services  are 
calculated.  An  output  is  defined  as  the  end  result  of  an  organization’s  operations;  it  can 
be  a  product  or  service  as  shown  in  the  diagram  below. 

As  a  performance  tool,  the  methodology: 

Compares  all  processes  in  terms  of  relative  productivity 

Allocates  revenues  to  common  units  of  output 

Measures  value  added  by  IT  by  the  outputs  it  produces 

Relates  outputs  to  cost  of  producing  those  outputs  in  common  units 
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Provides  common  unit  measures  for  organizational  productivity 

Based  on  the  tenets  of  complexity  theory,  KVA  assumes  that  humans  and  technology  in 
organizations  add  value  by  taking  inputs  and  changing  them  into  outputs  through  core 
processes  [Housel  and  Bell  2001].  The  amount  of  change  an  asset  or  process  produces 
can  be  a  measure  of  value  or  benefit.  Additional  assumptions  include: 

•  Describing  all  process  outputs  in  common  units  (i.e.,  the  time  it  takes  to  learn  to 
produce  the  required  outputs)  allows  historical  revenue  and  cost  data  to  be 
assigned  to  those  processes  at  any  given  point  in  time. 

•  All  outputs  can  be  described  in  terms  of  the  time  required  to  learn  how  to 
produce  them. 

•  Learning  Time,  a  surrogate  for  the  knowledge  required  to  produce  process 
outputs,  is  measured  in  common  units  of  time.  Consequently,  Units  of  Learning 
Time  =  Common  Units  of  Output  (K). 

•  Common  unit  of  output  makes  it  possible  to  compare  all  outputs  in  terms  of  cost 
per  unit  as  well  as  price  per  unit,  because  revenue  can  now  be  assigned  at  the 
sub-organizational  level. 

•  Once  cost  and  revenue  streams  have  been  assigned  to  sub-organizational  outputs, 
normal  accounting  and  financial  performance  and  profitability  metrics  can  be 
applied. 

Describing  processes  in  common  units  also  permits  market  comparable  data  to  be 
generated,  particularly  important  for  non-profit  organizations  such  as  the  DoD.  Market 
comparable  data  from  the  commercial  sector  can  be  used  to  estimate  price  per  common 
unit,  allowing  for  revenue  estimates  of  process  outputs  for  non-profits.  This  also 
provides  a  common  units  basis  to  define  benefit  streams  regardless  of  process  analyzed 
KVA  differs  from  other  ROI  models  because  it  allows  for  revenue  estimates  enabling 
useof  traditional  accounting,  financial  performance  and  profitability  measures  and 
prospective  financial  methods  as  real  options  analysis. 
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Figure  13:  Measuring  Output 


Traditional  Accounting/ 
Finance  Measure 

KVA  Process  Value  Measure 

Sales  /  Revenues 

Common  units  of  output 

Product  price 

Market  comparables:  Price  per  unit  of  output 

Total  Revenues 

Total  units  of  output  X  price  per  unit  =  total 
revenue  surrogate 

Table  4:  Comparison  of  Outputs 

Traditional  Accounting  Benefits  (Revenues)  versus  Process  Based  Value 


KVA  can  rank  processes  in  terms  of  the  degree  to  which  they  add  value  to  the 
organization  or  its  processes.  This  assists  decision-makers  to  identify  what  processes  are 
value-added  —  those  that  will  most  likely  accomplish  a  mission,  deliver  a  service,  or 
meet  customer  demand.  Value  is  quantified  in  two(IRM)/four(RO)  key  metrics: 
Return-on- Knowledge  (ROK)  for  KVA+IRM/RO,  Return  on  Knowledge  Assets  (ROKA) 
for  KVA+RO,  Return  on  Knowledge  Investment  (ROKI)  for  KVA+RO  and  Return  on 
Investment  (ROI)  for  KVA+IRM/RO. 

KVA  analysis  can  be  conducted  through  the  three  methods  shown  in  the  table  below. 
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Steps 

1 

2 


3 

4 

5 


6 


7 


Learning  Time 


Establish 
common  units  to 
measure  learning 
time 

Calculate  learning 
time  to  execute 
each  subprocess. 


Multiply  learning 
time  for  each 
subprocess  by 
number  of  times 
subprocess 
executes  during 
sample  period. 


Process  Description _ 

Identify  core  process  and  its  subprocesses. 
Describe  products  in  terms  of  instructions 
required  to  reproduce  them,  and  select  unit  of 
process  description. 

Calculate  number  of  process  instructions 
pertaining  to  each  subprocess. 

Designate  sampling  period  long  enough  to 
capture  representative  sample  of  core  process’s 
final  product/service  output. 

Multiply  number  of  process  instructions  used  to 
describe  each  subprocess  by  number  of  times 
subprocess  executes  during  sample  period. 


Allocate  revenue  to  subprocesses  in  proportion 
to  quantities  generated  by  Step  5,  and  calculate 
costs  for  each  subprocess. _ 

Calculate  ROK,  ROI,  and  interpret  results. 


Binary  Query  Method 


Create  set  of  binary  yes/no 
questions  such  that  ah  possible 
outputs  are  represented  as 
sequence  of  yes/no  answers. 
Calculate  length  of  sequence 
of  yes/no  answers  for  each 
subprocess. 


Multiply  length  of  yes/no 
string  for  each  subprocess  by 
number  of  times  this 
subprocess  executes  during 
sample  period. 


Table  5:  Approaches  to  KVA  Calculation  (Source:  Housel  &  Bell,  2001) 


Real  Options  Analysis 

Real  Options  analysis,  also  a  step  in  the  IRM  approach,  incorporates  strategic  planning 
and  analysis,  risk  assessment  and  management,  and  investment  analysis.  As  a  financial 
valuation  tool,  Real  Options  allow  organizations  to  adapt  decisions  to  respond  to 
unexpected  environmental  or  market  developments.  As  a  strategic  management  tool, 
Real  Options  are  a  strategic  investment  valuation  tool  affording  decision-makers  the 
ability  to  leverage  uncertainty  and  limit  risk. 

The  value  of  information  is  clear  after  the  uncertainty  is  resolved.  This  value  of 
information  is  called  as  the  ex-post  value  of  information.  The  ex-post  value  of 
information  is  just  the  difference  between  the  value  of  decisions  with  and  without  the 
information,  and  it  can  be  expressed  as  following  equation; 

VOI  =  Vix1)  -  R(x°) 
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where,  VOI  and  V  stands  for  the  value  of  information  and  the  value  of  decision, 
respectively,  xdenotes  the  decision  variable.  x;is  the  best  choice  of  the  decision  maker 
given  the  information  and  x°  stands  for  the  best  choice  without  the  information,  i.e., 

x1  —  argmaxE[F(x)|/] 
x°  =  argmax  E[V(x)] 

X 


whereE[-]  means  expected  value. 

When  the  information  is  perfect,  the  expected  value  of  the  information  can  be  calculated 
as  following.  Let  i  and  j  denote  the  choice  and  future  status,  respectively.  means  the 
probability  that  the  status  j  occurs,  stands  for  the  pay  off  that  the  decision  maker 
chooses  the  action  i  and  status  j  happens.  Then  the  expected  value  of  the  perfect 
information  (EVPI)  is 


EVP  I  =  2 ^  p7- (max  fly)  -  max^  VjRij 
j  j 

An  example  can  help  to  understand  the  mathematical  model  of  the  value  of  information. 
Following  table  shows  the  possible  future  economic  status;  boom  and  depression, 
feasible  investment  instruments;  stock  and  bond,  and  payoffs  according  to  the  status 
and  instruments. 


Stock 

Bond 

Boom 

20 

2 

Depression 

-10 

2 

Suppose  that  the  decision  maker  has  the  information  that  the  probability  of  boom  is 
50%  and  the  probability  of  depression  is  50%.  Then  the  expected  value  of  stock  is  5  and 
the  expected  value  of  bond  is  2.  Therefore  investing  to  stock  is  chosen  (x°  =  stock). 
Consider  the  decision  maker  obtains  the  information  that  the  probability  of  boom  is  o 
and  that  of  depression  is  1.  When  the  decision  maker  uses  this  information,  he  should 
choose  bond  (x1  =  bond).  If  the  real  future  situation  is  depression,  the  value  of  this 
information  is  F(xO  —  F(x°)  =  2  —  (— 10)  =  12.  On  the  other  hand,  if  the  future 
situation  is  boom,  the  value  of  this  information  is  2  —  20  =  —18.  Due  to  the  wrong 
information,  the  decision  maker  makes  an  undesirable  decision.  So  the  value  of  the 
information  is  negative. 

When  there  exists  the  perfect  information  about  future  status,  we  can  calculate  the 
expected  value  of  the  perfect  information.  Note  that  once  the  perfect  information  is 
acquired,  we  know  the  future  status  for  sure.  However  before  we  have  the  information 
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we  don’t  know  what  it  will  tell  us.  If  the  perfect  information  tells  that  future  status  is 
boom,  investing  to  stock  should  be  chosen.  Otherwise,  the  decision  should  be  bond. 
What  is  the  probability  that  the  perfect  information  tells  that  the  future  status  is  boom 
or  depression?  The  only  available  information  is  the  current  information.  Therefore,  the 
value  of  perfect  information  is 

EVPI  =  (0.5  x  20  +  0.5  x  2)  -  (0.5  x  20  +  0.5  x  -10)  =  11 

According  to  Hirshleifer  and  Riley  [1979],  the  value  of  information  is  an  outcome  of 
choice  in  uncertain  situations.  The  general  conclusions  from  models  of  information  are 
that  its  value  largely  depends  on  several  factors: 

•  How  much  flexibility  decision  makers  have 

•  The  value  of  output  in  the  market 

•  The  cost  of  the  information 

•  What  is  the  opportunity  cost  of  the  information 

The  first  factor  means  that  the  information  is  more  valuable  as  the  feasible  actions  are 
various.  Although  a  decision  maker  has  very  accurate  information,  if  he  has  no  choice 
but  doing  as  he  has  done,  the  information  is  useless.  The  second  factor  implies  that  the 
value  of  information  is  derived  from  the  demand  of  the  ultimate  output.  For  example, 
the  value  of  information  about  the  deposit  of  oil  depends  on  the  price  of  oil.  If  the  price 
of  oil  is  zero,  the  value  of  information  is  also  zero,  no  matter  how  the  information  is 
accurate. 


3.3.2  Quantitative  Evaluation  of  Flexibility  Using  Case  Studies 


SHIPMAIN :  The  Knowledge  Value  Added  +  Integrated  Risk  Management  (KVA+IRM) 
framework  is  used  to  quantify  process  improvements  and  the  potential  benefits  of  select 
technology  on  the  ship  maintenance  and  modernization  (SHIPMAIN)  program. 

SHIPMAIN  is  a  large  program  with  many  interrelated  concepts,  instructions,  policies, 
and  areas  of  study. 

SHIPMAIN  is  one  of  the  latest  initiatives  aimed  at  garnering  efficient  ways  to  get  the  job 
done.  It  is  a  best  business  practice  that  fleet  sailors  and  shipyards  are  utilizing  to 
change  the  culture  of  completing  ship  work.  The  Navy  implemented  the  SHIPMAIN 
process  in  FY  2004  to: 
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•  Increase  efficiency  of  maintenance  and  modernization  process  without 
compromising  effectiveness, 

•  Define  common  planning  process  for  surface  ship  maintenance  and  alterations, 

•  Install  disciplined  management  process  with  objective  measurements,  and 

•  Institutionalize  that  process  and  provide  continuous  improvement  methodology 
for  it.  (Commander,  Naval  Sea  Systems  Command,  2006) 

SHIPMAIN  is  about  doing  the  right  maintenance  at  the  right  time,  in  the  right  place  for 
the  right  cost.  The  initiative  seeks  to  identify  redundancies  in  maintenance  processes 
and  eliminate  them.  It  provides  a  single  process,  assisting  the  Navy  in  realizing  the 
maximum  benefit  per  maintenance  dollar  by  eliminating  time  lags,  prioritizing  ship  jobs 
and  empowering  Sailors  in  their  maintenance  decisions  (Commander,  Naval  Sea 
Systems  Command,  2006). 

The  SHIPMAIN  process  is  comprised  of  five  distinct  phasesi  and  three  Decision  Points 
(DP)2  to  take  a  proposed  change  from  concept  to  completion  in  one  document:  the  Ship 
Change  Document  (SCD).  (See  Appendix  for  more  details) 

KVA  Analysis:  As-is  Scenario:  A  summary  of  the  high-level,  As-is  KVA  analysis  is 
depicted  in  Table  6.  These  estimates  were  compiled  from  interviews  of  SMEs  at 
NAVSEA  and  from  historical  data  contained  in  the  NDE.  This  sample  is  representative 
of  availability  periods  for  ships  of  the  Pacific  and  Atlantic  Fleet,  including  Aircraft 
Carriers,  averaged  from  FY  2002  to  FY  2007.  All  estimates  contained  in  this  analysis 
are  as  conservative  and  accurate  as  possible. 

KVA  Results:  To-be  Scenario:  The  SHIPMAIN  process  was  reengineered  by  adding 
3D  laser  scanning  tools  and  a  comprehensive  suite  of  PLM  products  to  the  as-is  state. 
Implementation  of  3D  laser  scanning  tools  will  primarily  affect  Block  265.1  by  enabling 
the  planning  yard  to  acquire  images  and  output  its  drawings  in  a  highly  accurate  and 
electronically  transferable  3D  format— as  opposed  to  static  installation  drawings 
delivered  on  paper.  The  3D  scanning  tools  can  produce  a  2D  output  also,  as  currently 
required  under  the  FMP.  With  the  addition  of  a  robust  PLM  product  suite,  the  3D 
images  generated  can  be  shared  across  the  enterprise  in  an  Integrated  Data 
Environment,  allowing  all  stakeholders  real-time  access  to  highly  accurate  as-built 
imagery  through  a  single  interface. 


1Five  Phases:  I-Conceptual,  II-Preliminary  Design,  Ill-Detailed  Design,  IV-Implementation,  V-Installation  (Commander,  Naval  Sea 
Systems  Command,  2006). 

2  DPs  occur  at  the  conclusion  of  Phases  I-III.  Each  DP  is  an  approval  for  funding  of  successive  phases  and  has  an  associated  Cost 
Benefit  Analysis  (CBA),  Alteration  Figure  of  Merit  (AFOM)  and  Recommended  Change  Package  (RCP)  (Commander,  Naval  Sea 
Systems  Command,  2006). 
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As  Is  SHIPMAIN  Process  Overview 


Core  Process 

Process  Title 

Number  of 
Employees 

Total 

Benefits 

Total  Cost 

ROK 

ROI 

Block  250 

Authorize  and  Issue  Letter  of  Authorization 
(LOA)/Hull  Maintenance  Plan  (HMP); 
Generate  2Ks 

9 

$22,619,472 

$5,311,299 

426% 

326% 

Block  265 

Hull  Installation  and  Risk  Assessment 

44 

$94,928,918 

$130,071,059 

73% 

-27% 

Block  270 

Authorize  Installation 

4 

$24,710,347 

$3,161,555 

782% 

682% 

Block  280 

Resolve  "Not  Authorized/Deferred  SC 

1 

$3,706,552 

$619,523 

598% 

498% 

Block  300 

Install  SC 

46 

$94,722,998 

$40,617,720 

233% 

133% 

Block  310 

Feedback:  Cost,  CM,  Performance, 
Schedule,  ILS 

2 

$1,853,276 

$619,523 

299% 

199% 

Block  320 

Continue  Installs 

5 

$4,633,190 

$3,068,367 

151% 

51% 

Block  330 

Final  Install,  Closeout  SC 

1 

$926,638 

$309,762 

299% 

199% 

$248,101,392 

$183,778,809 

135% 

35% 

Table  6:  SHIPMAIN  Phases  IV  and  V  As-is  Core  Process  Model 


Implementation  of  an  enterprise-wide  PLM  product  suite  demonstrated  a  remarkable 
effect  on  each  core  process.  Providing  stakeholders  access  to  real-time  information 
related  to  all  iterations  of  the  product  lifecycle  in  a  collaborative  environment  enabled 
nearly  all  sub-processes  to  benefit.  Processes  that  didn’t  demonstrate  a  quantitative 
improvement  in  this  model  will  likely  show  qualitative  improvements  (which  will  be 
discussed  in  the  Conclusions  section).  Table  7  depicts  the  change  in  cost  and  ROI 
factors  from  the  As-is  to  the  To-be  scenario.  The  majority  of  the  estimates  contained  in 
this  table  were  derived  from  interviews  with  SMEs  from  NAVSEA  and  SIS  and  from  a 
comprehensive  review  of  the  business  rules  listed  in  Appendix  D  of  the  SCEPM  dated 
December  11,  2006. 

Results  shown  in  Table  7  demonstrate  that  overall  costs  would  be  reduced  by  nearly  $78 
million  dollars,  despite  additional  expenditures  of  acquiring  3D  laser  scanning  and  PLM 
tools.  It  is  apparent  that  cost  savings  are  achieved  in  all  processes,  with  the  exception  of 
Block  270,  as  a  result  of  3D  laser  scanning  and  PLM  tools.  As  the  technologies  mature, 
and  work  processes  are  modified  to  maximize  their  potential,  cost  savings  and  ROI 
should  continue  to  improve  over  time. 
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Core 

Process 

Process  Title 

Annual  As-ls 

Cost 

Annual  To-Be 

Cost 

Difference  (Cost 

Savings) 

As-ls 

ROI 

To-Be 

ROI 

Block  250 

Authorize  and  Issue  Letter  of 
Authorization  (LOA)/Hull  Maintenance 
Plan  (HMP);  Generate  2Ks 

$5,311,248 

$2,287,671 

$3,023,577 

326% 

565% 

Block  265 

Hull  Installation  and  Risk  Assessment 

$130,060,112 

$63,437,554 

$66,622,558 

-27% 

155% 

Block  270 

Authorize  Installation 

$3,161,600 

$3,217,805 

($56,205) 

682% 

668% 

Block  280 

Resolve  "Not  Authorized/Deferred  SC 

$619,424 

$427,964 

$191,460 

498% 

766% 

Block  300 

Install  SC 

$40,616,160 

$33,433,420 

$7,182,740 

133% 

183% 

Block  310 

Feedback:  Cost,  CM,  Performance, 
Schedule,  ILS 

$619,424 

$242,107 

$377,317 

199% 

665% 

Block  320 

Continue  Installs 

$3,068,520 

$2,510,944 

$557,576 

51% 

131% 

Block  330 

Final  Install,  Closeout  SC 

$309,712 

$304,059 

$5,653 

199% 

205% 

Totals: 

$183,766,200 

$105,861,524 

$77,904,676 

Table  7:  As-is  and  To-be  Cost  and  ROI  Value  Differences 


KVA  Real  Options  Analysis:  Real  Options  analysis  was  performed  to  determine  the 
prospective  value  of  three  basic  options  over  a  three-year  period  using  KVA  data  as  a 
platform.  The  Approach  involves  the  following  eight  procedural  steps: 

1.  Qualitative  management  screening 

2.  Forecasting  and  prediction 

3.  Base-case  KVA  net  present  value  and  ROI  analysis 

4.  Risk-based  Monte  Carlo  simulation 

5.  Strategic  Real  Options  problem  framing  and  courses  of  action 

6.  Real  Options  modeling  and  analysis 

7.  Analytical  portfolio  and  resource  optimization 

8.  Reporting  and  update  analysis 

After  running  the  different  scenarios,  “To  Be”  and  “Radical  To  Be”  provide  highest 
overall  total  strategic  value  with  little  difference  between  the  two  (19.51  to  20.49  times 
improvement  over  the  baseline  “As  Is”  option).  However,  when  considering  all  the 
downstream  options  available  from  collaborative  technologies  with  3D  scanning 
capabilities,  the  “Radical  To  Be”  course  of  action  is  the  best,  providing  an  overwhelming 
68.88  times  the  returns  from  the  existing  “As  Is”  base  case. 
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Maturity  (Years) 

5 

Risk-Free  Rate  (%) 

5.00% 

Strateqic  Option  Valuation 

AS-IS 

TO-BE 

RADICAL 

Benefits 

$ 

49,175,536.83 

$ 

93,344,192.00 

$ 

95,097,452.00 

Costs 

$ 

44,705,033.48 

$ 

7,854,206.09 

$ 

4,488,887.70 

Volatility 

N/A 

8.04% 

9.81% 

Total  Strategic  Value 

$ 

4,470,503.35 

$ 

87,227,330.00 

$ 

91,601,502.00 

Factor  Increase 

19.51 

20.49 

Maturity  (Years) 

10 

10 

10 

Factor  Increase 

3 

3 

10 

AS-IS 

TO-BE 

RADICAL 

Benefits 

$  147,526,610.48 

$280,032,576.00 

$  950,974,520.00 

Costs 

$  134,115,100.43 

$ 

23,562,618.26 

$ 

44,888,876.96 

Volatility 

N/A 

25.43% 

31 .02% 

Long  Term  Total  Strategic  Value 

$ 

13,411,510.04 

$265,742,275.00 

$  923,752,800.00 

Factor  Increase 

19.81 

68.88 

Table  8:  Summary  of  Results 


The  options  analysis  clearly  indicated  that  real  benefits  of  using  the  combined 
technologies  is  in  the  future  value  they  will  create  over  time.  While  the  cost  savings  from 
the  use  of  these  technologies  is  substantial,  the  real  story  is  in  the  future  value  these 
options  will  create  through  greater  flexibility  in  including  many  vendors  in  the  bidding 
process,  a  reduced  cycle  time  in  completing  the  maintenance,  and  the  possibility  of 
creating  “portless”  maintenance  for  ships  while  at  sea  or  in  foreign  ports. 

CCOPS:  KVA  methodology  was  applied  to  quantify  the  value  added  by  Cryptologic 
Carry-On  Program  (CCOP)  systems,  information  warfare/cryptologic  operators,  and  the 
enabling  ship  borne  system  infrastructure  with  which  they  interact.  Value  provided  by 
human  capital  elements  were  compared  to  IT  elements  to  measure  efficiency 
(productivity)  and  effectiveness  (profitability).  All  assets,  sub-processes,  and  outputs  are 
first  identified. 

•  Asset  analysis  encompasses  all  value  and  cost  data  related  to  each  asset  in  the 
process,  human  capital  or  IT  asset. 

•  Sub-process  analysis  includes  a  detailed  breakdown  of  the  ICP  to  include  the 
time-to-learn,  how  to  perform  each  sub-process,  and  number  of  executions  for 
each  sub-process. 

•  Process  outputs  are  established  via  time  to  learn  estimates,  including  the  total 
number  of  aggregated  process  outputs  and  a  surrogate  revenue  stream  used  to 
monetize  the  outputs. 
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Asset  values  and  costs  are  then  allocated  throughout  the  sub-processes  in  which  they 
contribute  to  the  production  of  outputs.  The  time-to-learn  (knowledge  embedded  in 
each  sub-process)  is  multiplied  by  the  number  of  executions  of  that  sub-process,  and  the 
figure  serves  as  a  basis  for  revenue  allocation  at  the  sub-process  level.  Costs  are 
calculated  by  multiplying  the  time  it  takes  to  produce  the  process  output  times  the 
salary  of  those  producing  it  and  the  cost  per  usage  of  the  IT  asset.  Costing  typically  does 
not  include  the  cost  of  fixed  assets  as  these  costs  are  typically  used  as  a  constant 
weighting  factor.  Therefore,  these  costs  usually  do  not  affect  the  relative  performance 
estimates  for  the  various  sub-processes.  Performance  ratios  such  as  ROKA  and  ROKI 
can  be  calculated  after  costs  and  benefits  for  each  sub-process  are  defined.  (See 
Appendix  for  more  details) 

KVA  Results:  KVA  analysis  was  used  to  compare  two  example  sub-processes:  “Search 
and  Collect”  (P4)  and  “Format  Data  for  Report  Generation”  (P8).  Results  are 
summarized  in  the  following  tables  and  issues  were  identified  at  the  portfolio,  program 
and  process  levels. 


Sub-Process 

CCOP  A 

CCOP  B 

CCOPC 

CCOP  D 

ROK 

Review  Request/Tasking 

PI 

168.54% 

168.54% 

Determine  Op/Equip  Mix 

P2 

166.86% 

166.86% 

Input  Search  Function/CoveragePlan 

P3 

152.91% 

152.91% 

Search/Collection  Process 

P4 

930.03% 

148.15% 

590.13% 

Target  Data  Acquisition/Capture 

P5 

290.15% 

147.71% 

228.23% 

Target  Data  Processing 

P6 

319.39% 

162.59% 

436.13% 

28.18% 

142.41% 

Target  Data  Analysis 

P7 

149.98% 

534.76% 

34.55% 

121.42% 

Format  Data  for  Report  Generation 

P8 

143.34% 

143.34% 

QC  Report 

P9 

315.88% 

315.88% 

Transmit  Report 

P10 

148.75% 

148.75% 

ROK  for  Total  Process 

278.59% 

152.81% 

485.44% 

31.37% 

196.27% 

Table  9:  Return  on  Knowledge  (ROK)  USS  READINESS  Summary  KVA  Results 


CCOP  D  is  a  cost -heavy  system  that  executes  very  few  times  with  negative  ROKs  throughout  the 
sample  period,  as  seen  in  Table  9. 
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•  Is  CCOP  D  appropriate  for  this  platform  and  mission? 

•  What  is  a  less  expensive  alternative  to  CCOP  D? 

•  Are  all  operators  appropriately  trained  in  the  use  of  CCOP  D? 


Sub-Process 

CCOP  A 

CCOP  B 

CCOPC 

CCOP 

D 

ROKI 

Review  Request/Tasking 

PI 

68.54 

22.11 

Determine  Op/Equip  Mix 

P2 

66.86 

20.89 

Input  Search 
Function/Coverage  Plan 

P3 

52.91 

-18.44 

Search/Collection  Process 

P4 

830.03 

48.15 

239.01 

Target  Data 
Acquisition/Capture 

P5 

190.15 

47.71 

47.28 

Target  Data  Processing 

P6 

219.39 

62.59 

336.13 

-71.82 

36.67 

Target  Data  Analysis 

P7 

49.98 

434.76 

-65.45 

21.25 

Format  Data  for  Report 
Generation 

P8 

43.34 

-20.37 

QC  Report 

P9 

215.88 

79.19 

Transmit  Report 

P10 

48.75 

-17.37 

Metrics  for  Aggregated 

178.59 

52.81 

385.44 

68.63 

109.9 

Table  10:  Return  on  Knowledge  Investment  (ROKI)  USS  READINESS  Summary  KVA  Results 


The  Search  and  Collect  process  (P4)  is  knowledge-intensive  requiring  IT  and  human 
capital  asset  investments  to  complete,  as  indicated  in  Table  10.  Moreover,  each  process 
output  necessitates  many  executions  of  the  sub-process. 

•  Could  an  even  higher  return  be  achieved  with  further  automated  search  and 
collection  systems  or  more  operators? 

•  Should  the  amount  of  knowledge  in  humans  and  IT  be  adjusted? 
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•  Could  a  broader  range  of  training  allow  operators  to  perform  more  functions? 

The  Search  and  Collect  process  (P4)  is  a  high  performer  with  an  overall  return  of  239% 
compared  to  a  -20.37%  return  for  the  Format  Data  for  Report  Generation  process  (P  8). 

•  What  accounts  for  the  discrepancy  in  the  returns  received  on  each  process? 

The  Format  Data  for  Report  Generation  process  (P  8)  only  executes  once  per 
intelligence  report  (process  output)  with  nearly  one  third  of  all  operators  assigned  to 
this  sub-process  one  fifth  of  the  total  human  cost. 

•  What  causes  this  low  efficiency  level? 

The  Format  Data  for  Report  Generation  process  (P  8)  is  more  automated  than  P4. 

•  Could  this  process  be  further  automated  or  performed  by  other  operators  to  yield 
higher  efficiency  and  effectiveness  levels? 

Real  Options  Analysis:  Real  options  analysis  was  performed  to  determine  the 
prospective  value  of  three  basic  options  over  a  three-year  period  using  KVA  data  as 
input  for  the  analysts.  Three  potential  scenarios  were  identified. 


Option  A 

Remote  to  Shore 

Option  B 

Direct  Support 

Option  C 

Permanent  SSES 

•  Data  viewed  from  geographically  remote  center. 

•  Intelligence  collection  processing  from 
consolidated  center  requires  less  intelligence 
personnel  on  ships. 

•  Consolidating  capabilities  into  central  center 
popular  movement  to  cut  costs  and  provide  more 
shore  based  operations  to  support  war-fighting 
capabilities. 

•  Similar  to  consolidation  of  service  operations  in 
businesses  into  larger  and  fewer  call  centers. 

•  CCOP  equipment  &  operators 
move  from  ship  to  ship 
whenever  a  ship  came  into  port 
for  maintenance,  repair  or 
modernization. 

•  Fewer  sets  of  CCOP  equipment 
and  operators  required  to 
service  intelligence  gathering 
needs  of  the  fleet. 

•  CCOP  systems  and  operators 
assigned  to  given  ships  at  all 
times. 

•  Requires  more  operators  and 
CCOP  systems. 

•  Potential  costs  increases, 
provides  more  control  of 
intelligence  capability  by  the 
ships  and  fleet  commanders. 

Table  11:  CCOP  Strategic  Scenarios 


Each  strategic  scenario  is  explored  further. 

Results  of  the  real  options  analysis  indicate  that  Option  C  delivers  the  highest  value  at 
$15.2  million.  Although  apriori,  Options  A  and  B  were  expected  to  have  significant  cost 
savings,  it  is  possible  to  see  greater  total  value,  with  much  lower  volatility  (risk),  for 
Option  C  with  RO  analysis.  Fleet  and  Ship  Commanders  who  intuitively  preferred 
Option  C  because  it  permitted  greater  control  of  intelligence  assets  for  specific 
operations,  now  have  objective  data  to  help  them  review  their  preferred  option.  This  is 
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not  to  say  that  the  other  options  might  provide  greater  strategic  value  in  the  long  run 
once  they  are  implemented  with  more  productive  CCOPs  assets  and  lower  volatility 
based  on  overcoming  the  initial  decrements  in  the  learning  curve  of  a  new  process 
implementation. 


Option  A 

Option  B 

Option  C 

PV  Option  Cost  (Year  1) 

$348,533 

$1,595,697 

$1,613,029 

PV  Option  Cost  (Year  2) 

$4,224,487 

$3,043,358 

$4,494,950 

PV  Option  Cost  (Year  3) 

$3,688,994 

$10,105,987 

$8,806,643 

PV  Revenues 

$24,416,017 

$33,909,554 

$38,820,096 

PV  Operating  Costs 

$16,220,188 

$16,765,513 

$9,951,833 

PV  Net  Benefit 

$8,195,829 

$17,144,041 

$28,868,264 

PV  Cost  to  Purchase  Option 

$425,000 

$169,426 

$72,611 

Maturity  Years 

3.00 

3.00 

3.00 

Average  Risk-Free  Rate 

3.54% 

3.54% 

3.54% 

Dividend  Opportunity  Cost 

0.00% 

0.00% 

0.00% 

Volatility 

26.49% 

29.44% 

15.04% 

Total  Strategic  Value  with 
Options 

$1,386,355 

$4,466,540 

$15,231,813 

Table  12:  Summary  Real  Options  Analysis  Results 


3.3.3  Strengths  and  Weaknesses 


Strengths 

•  Quantifies  value  of  specific  processes,  functions,  departments,  divisions,  or 
organizations  in  common  units  of  output. 

•  Provides  historical  data  on  costs  and  revenues  of  specific  processes  and  specific 
programs  within  organizations. 

•  Provides  a  methodology  that  will  facilitate  regulatory  compliance  in  the  public 
sector  with  legislation  such  as  the  Clinger-Cohen  Act  of  1996  mandating  portfolio 
management  for  all  federal  agencies.  In  the  private  sector,  it  can  facilitate 
compliance  with  Sarbanes-Oxley  by  making  performance  among  corporate 
entities  more  transparent. 

•  Highlights  operational  efficiencies/inefficiencies  at  any  level  of  analysis,  down  to 
individual  employees  and  IT  system. 

•  Leverages  current  and  future  portfolio  investments  by  estimating  the  potential 
total  value  created. 
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Weaknesses 

•  Financial  options’  assumptions,  such  as  no  arbitrage  condition,  complete  market 
condition  and  infinite  liquidity,  may  not  hold  for  the  non-financial  market. 

•  Without  checking  the  assumption  of  Black-Scholes  model,  using  the  Black- 
Scholes  formula  does  not  make  sense.  For  example,  the  strongest  assumption  of 
the  model  is  the  fact  that  uncertainty  can  be  modeled  in  geometric  Brownian 
motion  and  as  a  result  the  distribution  of  future  status  is  log-normal  distribution. 
If  the  future  environment  cannot  be  modeled  with  this  stochastic  process  and 
distribution,  the  Black-Scholes  model  is  not  valid. 

•  In  the  real  world  decision,  there  are  many  qualitative  characteristics  to  be 
considered.  In  the  real  option  approach  it  is  hard  to  be  considered. 

•  Almost  all  real  options  related  literature  assumes  the  risk-neutral  decision  maker 
implicitly  or  explicitly.  This  assumption  need  to  be  check  in  risk  management 
sense. 

•  With  the  raw  data  required  for  the  analysis  residing  in  multiple  databases  of 
varying  classification  levels,  data-gathering  mechanisms  that  are  less  human¬ 
intensive  and  more  automated  need  to  be  created  to  extract  the  required 
information. 

•  Although  the  ICP  in  this  case  study  was  developed  through  the  use  of  subject 
matter  experts,  a  standard  description  and  definition  of  each  sub-process  should 
be  reached  through  an  Intelligence  Community-wide  consensus  of  process  stake¬ 
holders. 

•  A  more  detailed  research  should  be  conducted  to  analyze  the  knowledge 
embedded  in  each  IT  system  to  accurately  capture  the  benefits  resulting  from  the 
execution  of  particular  system  processes. 

•  The  Market  Comparables  approach  to  valuing  the  outputs  of  non-profit 
organizations,  although  used  as  a  rough  baseline  to  monetize  outputs  in  this  case 
study,  requires  a  more  in-depth  look  at  comparable  organizations  utilizing 
similar  processes  to  produce  similar  outputs.  The  creation  of  a  broad  database  of 
such  organizations  is  currently  being  conducted  to  benchmark  industries  by 
functional  groupings  and  products. 

•  To  provide  a  more  powerful  analysis  of  the  ICP,  a  database  of  comparable 
historical  KVA  information  should  be  created  to  benchmark  future  work  or  to 
provide  a  broader  insight  for  current  work. 
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4  Research  Roadmap 


4.1  Gap  Analysis 

In  our  gap  analysis,  we  address  challenges  in  applying  the  MPTs  above,  and  also  do  a 
top-level  exploration  of  the  case  involving  different  sources  of  change.  Example 
challenges  include  foreseeable  but  uncontrollable  sources  of  change,  such  as  with 
externally  evolving  interoperating  systems,  and  modularizing  around  multiple  sources 
of  change.  For  example,  adding  a  new  data  input  will  require  coordinated  changes 
among  the  data  management  module,  data-entry  user  interface  module,  and  mission 
logic  module.  It  is  also  the  case  that  unforeseeable  changes  may  not  be  confined  to  a 
single  module.  Then  one  will  often  have  architecture -breakers  and  need  alternative 
approaches.  For  this  situation,  no  good  theoretical  basis  is  available,  and  projects  can  at 
best  put  together  various  individual  strategies  to  better  accommodate  unforeseeable 
change.  These  strategies  include  trend  analysis  and  autonomy;  user  programmability; 
value-based,  lean,  and  agile  methods;  and  evolutionary  acquisition  with  concurrent 
architecture  rebaselining. 

The  gap  analysis  also  identifies  the  major  risks  associated  with  MPTs  for  both 
achievement  and  valuation  of  flexibility,  primarily  for  the  foreseeable-change  case,  but 
also  considering  the  unforeseeable  change  case.  These  risks  include  difficulties  in 
achieving  scalability,  generality,  usability,  and  adversary-proofing  of  the  flexibility 
MPTs,  and  the  risk  of  overemphasizing  flexibility  and  creating  unacceptable  tradeoffs 
with  other  KPPs.  They  will  be  mitigated  where  possible  by  comparison  of  alternative 
valuation  methods,  but  primarily  addressed  by  the  gap  analysis  and  creation  of  a 
research  roadmap  for  addressing  them  at  higher  levels  of  scale,  realism,  and  adversary- 
based  evaluation. 

Real  options  have  garnered  much  attention  among  researchers  interested  in  extending 
the  analytical  results  and  computational  methods  associated  with  financial  options  to 
areas  of  application  well  outside  the  field  of  Finance.  There  is  now  a  large  literature 
devoted  the  theory  and  deployment  of  so  real  options  reporting  theory  and  practice. 
Nonetheless,  there  remain  many  open  questions  regarding  the  appropriate  use  of  of  real 
option  theory  in  practice.  We  wish  to  review  some  of  the  cautionary  considerations 
regarding  the  application  of  real  option  theory  in  the  context  of  military  procurement 
processes  and  the  valuation  of  flexibility.  To  this  end,  we  appeal  to  summary  concerns 
reported  in  the  open  literature  de  Neufville  (2002),  Hubalek  and  Schachermayer, 
(1999),  Wang  and  de  Neufville  (2005)  together  with  our  own  observations. 
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•  Real  options  must  be  described  in  terms  of  specific  technologies  and  the  systemic 
domain  in  which  they  are  to  be  developed.  Financial  analysis  alone  is  insufficient 
to  frame  real  options.  This  is  quite  difficult,  when  as  yet  undeveloped 
technologies  are  under  consideration. 

•  Financial  options  are  well-defined  contracts  that  are  tradable  and  individually 
valued,  while  real  options  are  not.:  real  options  have  no  contract-specified 
exercise  price  of  time.  The  usefulness  of  valuing  every  potential  program 
alternative  that  provides  flexibility  is  not  clear. 

•  In  military  procurement  programs,  previous  experiences  associated  with  the 
development  of  similar  technologies  are  not  necessarily  available.  Hence,  valuing 
real  options  on  the  basis  of  so  called  "comparables"  becomes  questionable 
because  of  the  absence  of  available  data 

•  Real  options  are  most  often  path-dependent.  Hence,  direct  applicability  of 
traditional  financial  options  methodologies  is  not  appropriate,  as  the  underlying 
stochastic  differential  equations  are  not  necessarily  available 

•  Real  options  in  military  acquisition  programs  are  likely  to  be  highly 
interdependent.  Traditional  financial  option  pricing  methods  fail  here,  again, 
because  underlying  stochastic  differential  equations  may  be  unattainable. 

•  In  military  procurement  programs,  there  may  be  no  justifiable  reason  to  accept 
the  "no  arbitrage  assumption".  In  this  case,  general  option  pricing  theory  breaks 
down. 

•  There  is  typically  no  quantitative  or  qualitative  reason  to  believe  the  real  options 
have  uncertainty  in  price  that  follow  Brownian  motion.  That  is,  unlike  in 
financial  markets  where  there  exist  both  quantitative  and  qualitative  analyses 
that  support  by  weak  convergence  in  measure  principles  that  suggests  a  limiting 
Brownian  motion  price  process,  there  is  typically  no  similar  reasoning  supporting 
the  assumption  of  Brownian  behavior.  Hence,  the  semi-martingale  arguments 
leading  to  the  principal  results  of  general  option  pricing  are  not  applicable. 

Effect  of  Errors  in  Uncertainty  Estimation:  What  happens  if  a  decision  maker’s 
estimation  about  future  uncertainty  is  wrong?  What  is  the  damage  of  abusing  Black  - 
Scholes  model?  Maximum  entropy  principle  provides  an  important  implication  to  these 
questions.  The  probability  distribution  that  best  describes  the  current  information  is  the 
distribution  maximizing  the  information  entropy.  In  other  words,  when  we  have  testable 
information,  the  true  probability  distribution  with  respect  to  the  current  information 
maximizes  the  entropy.  The  principle  was  first  illustrated  by  E.T  Jaynes  in  1957. 

For  a  discrete  random  variable  with  distribution 

P( X  =  xk)=pk,  for  k  =  1,2,- 

the  entropy  of  X  is  defined  as 
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For  a  continuous  random  variable  X  with  probability  density  function  p(x),  the  entropy 
of  X  is  defined  as 


HO 0  =  - 


£ 


V  O)  log 


pOO 

m(x) 


dx 


Where, m(x)  is  called  invariant  measure. 

The  testable  information  is  the  statement  of  current  information  to  determine  whether 
or  not  a  given  distribution  is  consistent  with  it.  The  testable  information  plays  a  role  of 
constraints  of  the  maximization  problem.  For  discrete  case,  maximum  entropy 
probability  distribution  is  derived  with  following  procedure.  Let  /  be  testable 
information  about  a  quantity  x  which  takes  values  in  {x1,x2,  •••  ,xn}.  Suppose  that  we 
have  m  testable  information  of  the  distribution  which  is  represented  as 

n 

^P(Xi|/)/fc(Xi)  =  Fk,  k  =  l,-,m 

i=l 

Moreover,  the  sum  of  probabilities  should  be  l.  Therefore 

n 

^]p(Xi|/)  =  l 
i= 1 

With  these  constraints,  the  probability  distribution  with  maximum  information  entropy 
is 


F(Xi\0 


1 

— - —  expUiACXt)  +  •"  +  Am/mCXi)] 

A  fAi,  ,  A-m) 


Where,  Z(A1,  ■■■ ,  Am)  -  Yu=i  exP[^ifi(xd  +  •"  +  ^mfmixd\  an(i  the  parameters  Ak 
determined  by  the  constraints  Fk  =  —  log Z(AV  ••• ,  Am). 


For  continuous  distribution,  some  testable  information,  /,  about  x  takes  values  in  a 
interval  [a,  b] .  This  information  is  expressed  in  constraints  on  the  expectations  of  the 
function  fk. 

p(x\I)fk(x)dx  —  Fk,  k  —  1,  •••  ,m 
J a 

The  obvious  constraint  is  given  by 
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00 

I  p(x|/)dx  =  1 

j  —  CO 


The  probability  distribution  that  maximize  the  entropy  of  p(x)  subject  to  the  constraints 
is 


P(xt|/) 


1 

— - prm(x)  expUiACxi)  +  •••  +  Tm/m(xi)] 

Z  L/L1;  •••  ,Am) 


Where  Z^,  —  ,Am)  =  f  m(x)  ewp[A1f1{xi')  H - F  /bn/m(xt)]  dx.  Similar  to  the  discrete 

case,  the  coefficients  A1;  ••• ,  Am  are  determined  by  the  constraints 


Fir  = 


dA, 


log  Z  ••• ,  An j) 


Example 

Suppose  that  a  random  variable  x  takes  a  value  from  {0,1,2,  •••  ,100}.  A  decision  maker  is 
considering  to  implement  a  flexible  system  that  makes  it  possible  to  choose  system  A,  B 
or  C  according  to  the  realization  of  x.  The  payoff  of  system  dis  A  —  — x  +  30.  The  system 
B  pays  B  —  — (x  —  30)(x  —  70).  The  C  —  x  —  70  represents  the  payoff  of  the  system  C. 
Figure  14  shows  the  payoff  of  each  system 


Let  V*(x )  be  the  value  of  optimal  choice  when  the  future  state  is  x.  Then  V*(x)  can  be 
expressed  as 
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(  —  x  +  30  0  <  x  <  30 

17*  (x)  —  j — (x  —  30) (x  —  70)  30  <  x  <  70 

(  x  —  70  70  <  x 

What  is  the  value  of  this  flexible  system?  The  value  of  the  system  depends  on  the  future 
uncertainty.  We  can  express  the  expected  value  of  the  system  as 

100 

^P(x  =  i)  •  V*(i) 

i= 0 


Suppose  that  we  have  the  information  that  in  the  future  P(x  <  30)  =  0.1,  P(x  <  50)  = 
0.4  and  P(x  <  80)  =  0.8. 

Binomial  distribution  assumption:  If  the  decision  maker  assumes  that  the  random 
variable  x  follows  binomial  distribution,  the  distribution  parameters  can  be  estimated 
with  least  square  method  as  follows. 

min[{F^(30;  100,  p)  -  0.1}2  +  {F^(50;  100, p)  -  0.4}2  +  {Fbi( 80;  100, p)  -  0.8}2] 

p 

where,  Fbi(k ;  n,  p)  stands  for  the  cumulative  distribution  of  binomial  distribution  up  to  k 
with  parameters  n  and  p,  i.e.,  Fbi(k-,n,p )  =  Y*x=o  Q)p*(l  -  v)n~x- 

The  estimation  result  is  p*  —  0.5176.  The  expected  value  of  the  flexible  system  under  the 
binomial  distribution  assumption  is 

100 

^  (1°°)  0.5  1  76*(0.48  24)100-x  •  W(V)  =  371.94 

x=o 

Geometric  distribution  assumption:  The  decision  maker  may  assume  that  the 
random  variable  x  follows  exponential  distribution,  since  x  takes  positive  integer  value. 
With  the  similar  estimation  procedure  of  binomial  distribution  case, 

min[{Fex(30;  p)  -  0.1}2  +  {Fe*(50;  p)  -  0.4}2  +  {Fex( 80;  p)  -  0.8}2] 

p 

where,  Fex(k-,p )  =  1  —  (1  —  p)k+1  stands  for  the  cumulative  distribution  of  exponential 
distribution.  The  estimation  result  p*  —  0.0117.  The  expected  value  of  the  flexible 
system  under  the  geometric  distribution  assumption  is 
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100 

^  0.0117  x  0.9883*  •  E*(x)  =  86.24 

x=o 


Maximum  Entropy  probability  distribution:  The  testable  information  can  be 
rewritten  as  following  constraints. 

100 

2>(x  =  i)  •  ix<30  =  o.i 

i= 0 
100 

^  P(x  =  0  •  ^<50  =  0.4 
£= 0 

100 

^  P(x  =  0  •  L,<80  =  0.8 

i= 0 

We  can  invert  these  constraints  to  followings  for  convenience  of  calculation. 

100 

^  P(x  =  0  •  ^<30  =  o.l 

i= 0 
100 

P(x  =  i)  •  n31<x<50  =  0.3 

i=0 

100 

P(x  —  i)  •  Hsi<x<80  —  0.4 

i= 0 

100 

P(x  =  0  =  1 

i= 0 


fl(.xi)  —  ^Xj<30 !  ^1  =  0-1)  ^2  (X[)  —  I31<X<50j  ^2  =  0.3,/3(Xj)  =  H51<x<80j  ^3  =  0-4,  /iCXj)  = 
1  and  F4  —  1, where  H  represents  the  indicator  function.  The  maximum  entropy  function 
with  these  constraints  are  known  as  a  piecewise  uniform  distribution. 
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Z(AllA2,A3lA4)  —  31  exp[A4  +  A4\  +  20  exp[A2  +  A4]  +  30  exp[A3  +  A4\  +  20exp  [A4] 
d  1 


dAx 

d 

dA~2 

d 

~dAx 


Z(A4,  A2,  A3,  A4)  — 
Z(A1,A2,A3,A4)  — 
Z(A1iA2iA3,A4)  — 


dA± 


Z(Ai,A2fA3fA4)  — 


Z  (Alf  A2,  A3,  A4) 
1 

Z  (A4,  A2,  A3,  a4) 
1 

Z  (A1;  A2,  A3,  a4) 
1 


Z  (Alf  A2,  A3,  A4) 


31  exp[A4  +  A4]  =  0.1 
20  exp[A2  +  A4]  =  0.3 
30  exp[A3  +  A4\  —  0.4 
20  exp[A4]  =  1 


P(x  =  0)  = 


Z  (Alf  A2,  A3,  A4) 
1 


exp(A4)  =  —  Z(AlfA2>A3fA4) 


exp(Ai)  =  — 
exp(A2)  =  0.3 

expa3)  =  4 

exp[A4  +  A4]  = 


Z  (A1;  A2,  A3,  A4) 


exp[A4]  exp[A4] 


Z(A1,A2,A3,A4)  6XP^^  20  20  31  31 

The  maximum  information  entropy  distribution,  which  is  sometimes  called  the  Gibbs 
distribution,  is  piecewise  uniform  distribution  as  following 


p(x)  —  < 


(0.1 

31 

x  —  0,1,  •• 

•30 

0.3 

20 

x  —  31,32, 

••,50 

0.4 

30 

x  —  51,52, 

bo 

o 

0.2 

^20 

x  =  81,82, 

•,100 
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100 

^p(x)  -V*(x)  =  157.68 

x=o 

In  summary,  according  to  the  assumption  of  the  future  uncertainty,  the  evaluation  of 
flexibility  can  be  different.  As  this  simple  example  shows,  under  the  binomial 
assumption,  the  value  of  flexibility  is  overestimated  about  two  times.  On  the  other  hand, 
under  the  geometric  distribution  assumption,  a  decision  maker  may  underestimate  the 
value  of  flexibility  only  half  of  its  value. 

We  are  currently  working  on  developing  a  test  bed  based  on  these  principles  using  a 
defense  satellite  domain  case  study. 


4.2  Developing  a  Framework  for  Valuing  Flexibility 

Considering  the  current  state-of-the-art  and  the  gaps  found  in  the  current  capabilities  to 
value  flexibility,  we  are  working  towards  developing  an  analytically  rigorous  framework 
for  defining  and  valuing  flexibility  that  adheres  to  the  mathematical  foundations  of 
characterizing  uncertainty  and  fundamental  tenets  of  rational  decision  making  under 
uncertainty. 

A  candidate  value-based  definition  of  flexibility  is,  “A  system  is  flexible  to  the  extent  that 
it  can  be  cost-effectively  modified  to  meet  new  needs  or  to  capitalize  on  new 
opportunities.” 

In  this  context,  “Cost”  includes  dollars,  calendar  time,  critical  skills,  and  other  scarce 
resources  (facilities,  equipment,  supplies,  etc.).  It  also  includes  the  costs  of  flexibility- 
induced  decrements  in  other  system  attributes  (performance,  security,  safety,  usability, 
etc.). “Effectiveness”  includes  improvements  in  military  outcomes  across  a  range  of 
weighted  scenarios,  and  cost  avoidance  (e.g.,  cost  of  delay). 

Some  implications  of  this  definition  are  that  the  value  of  flexibility,  V(F),  will  vary  by 
mission  and  by  range  of  scenarios.  Thus,  there  is  no  one-size-fits-all,  silver  bullet  V(F) 
formula.  Also,  V(F)  will  vary  by  its  impact  on  other  system  attributes.  Examples  of 
flexibility-induced  decrements  are: 

•  With  performance:  loose  vs.  close  coupling  (supercomputing) 

•  With  development  cost  and  schedule:  more  to  design,  develop,  V&V  (rapid 
fielding) 

•  With  maintainability:  more  side  effects  to  address  (automotive) 

•  With  usability:  too  many  options  (Office  2007,  TV-DVD-VCR  wand) 

•  With  security:  too  many  entry  points  (Windows) 
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The  following  discussion  provides  our  initial  work  on  formalizing  the  value  based  notion 
of  flexibility.  This  framework  will  be  refined  and  operationalized  in  the  second  part  of 
this  project. 

Analytical  Model:  The  flexibility  of  a  system  should  be  analyzed  in  two  spaces, 
capability  space  and  need  space,  and  their  interdependent  relationship 

Capability  Space:  Capability  refers  to  the  ability  of  a  system  to  perform  a  task.  For 
example,  an  airplane’s  capability  consists  of  speed,  load  and  range.  Suppose  that  Qc  is 
the  sample  space  of  capability.  We  can  define  sigma-field  on  the  capability  space.  This 
sigma  field  contains  enough  information  for  decision  maker.  Let  oc  denote  the  sigma 
field  of  the  sample  space  of  capability.  For  example,  when  a  component  of  capability 
space  is  expressed  in  real  number,  Borel  set  can  be  enough  information  set  for  the 
capability.  If  a  component  has  finite  number  of  feature,  power  set  of  the  component  set 
can  be  the  information  set. 

Capability  space  is  defined  as  a  tuple  of  the  sample  space  and  the  information  set  of 
capability,  i.e.,  (nc,  oc ).  Every  system  has  a  representation  on  the  capability  space. 


Speed 


A 


u 


ABC 


> 


Type  of  Weapons 


Fighter 

Helicopter 


Figure  15:  Capability  Space  Representations  of  a  Fighter  and  a  Helicopter 


Figure  15:  shows  an  example  of  capability  space  representation  for  a  traditional  fixed 
wing  aircraft  and  a  helicopter.  Suppose  that  a  decision  maker  interested  in  aircrafts. 
Among  other  features,  the  decision  maker  focuses  on  2  features,  the  speed  of  aircraft 
and  the  weapon  which  can  be  installed  to  the  aircraft.  X  and  Y  axis  represents  the  speed 
of  aircrafts  and  the  type  of  weapons,  respectively.  The  fighter  can  fly  fast,  but  it  requires 
minimum  speed  to  fly.  On  the  other  hand,  the  helicopter  is  able  to  do  hovering,  although 
it  can’t  fly  at  supersonic  speed.  With  respect  to  usable  weapons,  above  picture  shows 
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that  a  fighter  can  use  B  and  C  types  of  weapons,  and  a  helicopter  can  use  A  and  B  types 
of  weapons. 

Note  that  the  dimension  of  the  capability  space  reflects  the  decision  makers’  interest. 
Suppose  that  a  decision  maker  consider  n  aspects  of  the  system.  Then  the  sample  space 
of  capability  is  n-fold  of  sample  space  of  each  aspect.  Let  flf  be  the  sample  space  of  £th 
feature  of  the  system  capability,  for  i  =  1,2,  •••  ,n.  Then  the  sample  space  of  the  system  is 
expressed  as  Qc  —  x  x  •••  x  fl%  .  In  the  same  way,  let  erf  denote  the  sigma  field  of 
£thaspect.  Then,  oc  =  x  o%  x  ■■■  x  0%.  A  system’s  capability  is  configurable  in  the 
capability  space.  It  means  that  a  decision  maker  can  find  a  representation  of  a  system 
(di)  on  a  capability  space  such  that  a>c  E  oc 

Need  space:  Need  space  represents  the  tasks  that  a  system  may/would  be  required  to 
perform.  This  space  depends  on  the  end-users  needs  for  the  specific  system,  which  may 
change  over  time.  Capability  space  emphasized  the  engineering  and  manufacturing 
aspects  of  the  system.  On  the  other  hands  need  space  represents  a  user  oriented  point  of 
view. 

Let  be  the  sample  space  of  need,  i.e.,  flw  stands  for  the  set  of  all  possible  needs  which 
can  be  satisfied  with  a  system.  crwmeans  the  information  set  of  needs.  Similarly  to  the 
capability  space,  the  pair  (nw,erw)  is  the  need  space.  The  dimension  of  the  capability 
space  is  determined  by  the  decision  makers’  interest.  Suppose  that  there  are  m 
characteristics  of  need  that  is  important  to  the  decision  maker.  Then  x  nf  x 

•••  x  .  In  the  same  way,  let  erf  denote  the  sigma  field  of  £thaspect.  Then,  oN  -  erf  x 
erf  x  •••  x  erf .  A  need  that  is  satisfied  with  a  system  is  represented  on  the  need  space  as 
o>N,  i.e., 

coN  E  aN 


The  need  is  closely  related  to  the  capability,  and  the  relationship  is  expressed  in  a 
measurable  function  from  capability  space  to  need  space.  Let  g  be  the  measurable 
function,  then  g\oc^oN.  Therefore  the  relationship  between  the  capability  space 
representation  (coc)  and  the  need  space  representation  (eow)  is  00 N  =  g(ooc) 

Since  the  function  g  is  measurable,  a  decision  maker  can  find  an  appropriate  capability 
when  a  need  is  given,  i.e.,  coc  —  ,g-1(<Dw). 
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Capability  Space  Need  Space 


Figure  16:  System  Analysis 


In  previous  example,  we  see  the  capability  space  representation  of  fighter  aircraft  and 
helicopter.  Figure  16  shows  the  relationship  between  the  capability  space  and  need 
space.  The  ‘force  projection’  and  ‘lethality’  are  the  examples  of  the  needs  that  those 
planes  satisfy.  Based  on  the  capability  representation  (coc),  a  decision  maker  can  find 
the  need  that  a  system  satisfies,  (<uw).  The  blue  circle  and  the  red  circle  on  the  need 
space  means  the  need  that  a  fighter  aircraft  and  that  a  helicopter  satisfies,  respectively. 
A  decision  maker  can  find  these  needs  by  analyzing  the  capability  representations  of  the 
planes. 


Capability  Space  Need  Space 


Speed  ' 

N  , 

Force 

Projection 

\ 

7 

^  *"  Type  of  Weapons 

Figure  17:  System  Design 

lethality 

Its  reverse  is  also  possible.  Figure  17  represent  that  the  reverse  procedure.  Suppose  that 
a  decision  maker  want  a  system  that  satisfies  the  need  represented  by  the  blue  circle  on 
the  need  space.  Then  we  can  figure  out  what  capability  is  needed  to  satisfy  the  desired 
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need.  Remark  that  the  capability  representation  that  satisfies  a  need  may  not  be  unique. 
In  figure  17,  two  different  systems  can  satisfies  the  required  need. 

Cost  function:  Cost  function  is  defined  on  the  capability  space  and  represents  the  cost 
of  achieving  the  capability  of  system.  Cost  includes  dollars,  calendar  time,  critical  skills, 
and  other  scarce  resources  such  as  facilities,  equipment  and  supplies.  It  also  includes 
the  costs  of  flexibility-induced  decrements  in  other  system  attributes  for  instance 
performance,  security,  safety  and  usability. 

The  characteristics  of  cost  are  important  information  about  the  fits  between 
environment  and  flexible  system.  For  instance,  suppose  that  architecture  is  very  low  cost 
in  a  certain  range,  but  dramatically  rise  beyond  the  range.  Then  the  architecture  is 
suitable  only  when  the  possibility  that  out  of  range  situation  is  occurred  is  very  low. 

How  the  cost  can  be  measured  has  been  a  formidable  task.  Even  in  net  present  value 
method,  one  of  the  most  famous  decision  rules,  estimating  future  cash  flow  including 
cost  is  still  controversial.  The  research  of  total  cost  of  ownership  will  be  helpful  to  clarify 
how  to  measure  the  cost. 


y(-)  | 


Future  uncertainty  on  need  space:  Future  uncertainty  can  be  recognized  on  the 
need  space  rather  than  the  capability  space.  For  example,  when  we  can  find  the 
probability  distribution  about  future  status,  the  probability  distribution  is  defined  on 
the  need  space.  Let  P(u>w)  denote  the  probability  distribution  function. 

Current  models  of  the  flexibilities  depend  on  the  assumption  about  future  uncertainty. 
For  example,  Black-Scholes  model  assumes  the  geometric  Brownian  motion  of  its 
stochastic  process  and  corresponding  lognormal  distribution  of  future  statues.  What  if 
the  process  is  not  the  geometric  Brownian  motion?  What  if  test  and  sensitivity  analysis 
may  provide  valuable  information  to  decision  makers.  In  this  sense,  imposing  general 
assessment  of  the  future  uncertainty  is  a  merit  of  this  analytical  model.  To  illustrate  this 
merit,  an  example  of  maximum  entropy  analysis  will  be  posted  on  the  appendix  of  this 
report. 
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Value  function:  Value  function  is  a  function  from  need  space  to  a  value  space. 
Generally  we  can  express  the  value  of  system  in  terms  of  real  number.  Then  the  value 
function  V  is  a  function  from  need  space  to  the  real  line,  i.e.,  V\oN  ■-»  E. 

Since  the  future  uncertainty  generates  the  value  of  flexibility,  the  expected  value  of 
system  is  important.  For  well  defined  value  function  and  probability  distribution 
function,  F(u>w)P(u>w)  denotes  the  expected  value  of  system. 

The  structure  of  analytical  model:  The  Figure  19  illustrates  the  structure  of 
analytical  model.  A  system  has  a  representation  on  the  capability  space.  The  cost 
function  is  defined  on  the  capability  space.  Based  on  the  capability,  a  decision  maker  can 
recognize  the  need  that  can  be  satisfied  by  the  system.  Moreover,  when  a  certain  need 
should  be  satisfied,  a  decision  maker  can  figure  out  what  capability  is  required  to  satisfy 
it.  Future  uncertainty  can  be  recognized  on  the  need  space.  The  need  determine  the 
value  of  the  system. 

The  analytical  model  is  a  general  model  of  existing  flexible  models,  such  as  modular 
system  model,  real  options  and  value  driven  design  approach.  In  later  section,  we  will 
see  how  the  existing  flexibility  models  are  interpreted  in  this  framework. 


Definition  of  Flexibility:  Flexibility  is  the  ability  adjusting  capability  efficiently 
according  to  the  change  of  environment.  When  a  system  changes  its  capability  from  A  to 
B,  its  flexibility  is  defined  as 


Flexibility  — 


ju(AAS) 

y(AAB) 
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where^(-)  and  y(-)  means  the  measure  of  capability  and  cost  function,  respectively. 
AABmeans  symmetric  difference  of  A  and  B,  i.e.  AAB  —  {x:  x  e  A  U  B  and  x  g  An  B}. 

The  flexibility  has  useful  properties.  The  flexibility  is  a  non-negative  real  number, 
0  <  flexibility  <  oo.  Intuitively,  a  negative  valued  flexibility  does  not  make  sense,  and 
this  property  catches  the  intuition.  As  the  value  of  flexibility  is  low,  the  system  is 
inflexible.  When  the  change  of  capability,  AAB,  is  o,  the  flexibility  is  o.  Suppose  that 
changing  the  current  capability  of  system  costs  enormously.  Then  the  flexibility  is  small. 
On  the  other  hand,  when  the  change  of  capability  is  big  and  the  cost  of  change  is  low,  the 
flexibility  increases. 

To  avoid  infinitely  big  flexibility  case,  assume  that  y(AAS)  >  0,  VAAB  s.  t.  fi(AAB)  >  0. 
Let’s  consider  the  product  family  design  in  terms  of  the  analytical  model  frame  work. 
When  performances  envelop  is  expanded  by  product  family  design,  the  flexibility  is 
increased  by  increasing  of  numerator.  Figure  20  shows  how  real  option  can  be 
interpreted  in  this  definition  of  flexibility.  Real  option  is  a  right  to  change  the  capability 
in  the  future,  therefore  it  increases  the  capability  difference  ^.(AAB)  and  enhances  the 
flexibility. 


Figure  20:  Real  Option 


Cost-down  methods  improve  the  flexibility.  A  typical  example  is  a  modular  system.  As 
shown  in  Figure  21,  we  can  easily  acquire  new  capability  with  modular  system.  This 
advantage  is  represented  as  the  lower  cost  of  required  capability,  comparing  to  the 
normal  system.  Since  the  modular  system  lower  the  denominator,  y(A AS),  it  increases 
the  flexibility. 
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Value  of  Flexibility:  The  value  of  flexibility  is  defined  by  the  difference  between  the 
value  of  the  systems  which  can  be  attainable  and  the  cost  to  change  the  capability 

V(g(AU  B))  -y(AAB) 

Note  that  g  (A  U  S)  means  the  need  that  is  satisfied  with  the  flexible  system. 

Suppose  that  current  system  is  A.  The  expected  value  of  flexibility  can  be  expressed  as 

p (g(A  u  5))  •  V(g(A  U  B))  -  P(g(A\B))y(AAB) 

Where  A\B  —  {x:  x  £  A  and  x  £  Bj 

Further  discussion  about  cost  function:  When  the  cost  of  change  is  not 
symmetric,  we  need  to  consider  it.  y(co afy)stands  for  the  cost  to  change  the  capability 
of  system  from  cof  to  This  cost  function  may  be  asymmetric.  In  other  words, 
y(eof,eo%)  * 

Suppose  that  a  new  need  should  be  satisfied  through  altering  the  system.  Let  co0  be  the 
current  capability  of  the  system.  A  decision  maker  finds  that  both  system  co1  and  co2  will 
be  good  for  new  need.  Then  what  system  should  the  decision  maker  choose?  If 
Y(cn0,  (Ux)  <  y(u>0,  u>2),  then  a*!  would  be  the  best  choice,  otherwise  ca2  would  be  the  best. 
Suppose  that  there  are  n  possible  system  changes,^,  u>2,  ••• ,  o)n,  to  satisfy  the  new  need. 
Then  the  rational  change  of  system  (coR)  is  defined  as 

toR  =  arg  min  y(w0,  cofy 
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4.3  Phase  II  Tasks  and  Research  Deliverables 

The  primary  goal  of  Phase  I  is  to  evaluate  current  flexibility  valuation  approaches  and 
identify  the  limits  of  existing  methods.  To  achieve  these  goals,  RE-18  research  team 
reviewed  existing  methods,  processes  and  tools  for  flexible  during  Workshop  I.  As  a 
result  of  this  analysis,  the  RT-18  team  found  that  the  existing  approaches  provide  useful 
tools  to  value  flexibility.  However,  these  approaches  have  several  assumptions 
embedded  in  their  formalism.  These  assumptions  may  not  apply  to  the  DoD  system 
design  and  acquisition  process.  The  existing  approaches  are  not  sufficient  to  answer  the 
frequently  asked  questions  such  as  “how  can  I  value  flexibility  when  the  information 
about  future  uncertainty  is  limited?"  or  ”how  can  I  trade-off  between  flexibility  and 
other  performance  measures?"  Analysis  of  past  cases  confirms  the  inadequacy  of 
present  methods.  Moreover,  the  valuation  tools  for  flexibility  have  been  developed  for 
specific  situations.  It  is  difficult  for  a  decision  maker  to  answer  the  question  ’Which  tool 
is  appropriate  in  my  situation?"  Therefore,  the  need  exists  to  define  a  comprehensive 
analytical  formalism  for  defining  and  valuing  flexibility  that  is  better  suited  for  use  by 
decision  makers  in  a  DoD  context. 

In  the  next  phase  of  the  project,  the  RT-18  team  will  develop  an  analytical  framework  to 
assess  the  value  of  flexibility.  Based  on  the  model,  the  RT-18  team  will  develop  an 
exemplary  case  study,  implementing  tools  developed  during  the  project.  Each  activity 
will  be  executed  in  complementary  manner  to  each  other.  For  example,  the  analytical 
model  provides  the  basis  of  case  study,  and  the  findings  from  case  study  will  be  used  to 
revise  the  model.  Through  these  activities,  the  RT-18  project  is  expected  to  help  decision 
makers  in  a  concrete  manner.  The  Following  figure  is  the  roadmap  of  the  RT-18 
research  project. 


TAMU 


AFIT 


use 


UVA 


NPS 


1 

Phase  1 

Gap 

Analysis 

Anecdotal 

Study 

Analytical 
.  Model 

Reviewing 

Tools 

Phase  II 


Analytical  Model 
Case 


Analytical  Model 
Case  Study 
Developing  Tools 


Developing  Tools 


Figure  22:  Research  Roadmap  of  RT-18  Project 
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Although  every  team  member  will  share  their  ideas  intensively  to  contribute  to  this 
research  project,  each  group  will  have  one  or  more  main  domains  to  maximize  the  merit 
of  the  diversity  of  the  RT-18  research  team. 

Action  Items 

•  Case  studies 

o  Structured  as  a  decision  tree 
o  What  data  is  available? 
o  What  are  the  options? 

•  Methods 

o  What  are  the  inputs/outputs? 
o  What  does  it  measure? 

•  Analytical  framework  refinement 
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5  Summary 


Despite  its  ubiquity  in  the  literature,  flexibility  remains  an  ambiguous  concept.  There 
exist  a  multitude  of  definitions,  which  vary  not  only  by  domain,  but  within  domains  as 
well.  Furthermore,  these  definitions  often  conflict  with  one  another,  making  it  difficult 
to  discern  the  intended  meaning  in  a  given  study,  and  nearly  impossible  to  form 
generalizations  across  studies.  Making  matters  worse,  there  is  a  plethora  of  related 
terminology  that  is  often  used  carelessly  and/or  interchangeably  with  flexibility,  thereby 
serving  to  further  obfuscate  its  meaning. 

Though  the  definition  of  flexibility  may  be  convoluted,  it  remains  highly  sought  after  in 
system  design.  Flexibility  is  believed  to  provide  a  host  of  benefits,  to  include  reducing 
risk,  decreasing  cost,  improving  responsiveness,  enhancing  capabilities,  and  extending 
system  life.  The  literature  also  indicates  a  strong  positive  correlation  between 
uncertainty  and  the  value  of  flexibility.  Given  these  ostensible  benefits,  and  given  that 
uncertainty  is  present  to  some  degree  in  all  acquisition  programs,  flexibility  is 
increasingly  viewed  as  a  potentially  powerful  tool  to  mitigate  those  sources  of 
uncertainty  that  perennially  plague  defense  acquisition,  such  as  changing  requirements, 
unstable  budgets,  shifting  technologies,  and  fluctuating  policies.  There  is  far  less 
discussion,  however,  regarding  the  downsides  of  flexibility.  Infusing  flexibility  into  the 
design  of  a  system  is  bound  to  require  some  type  of  investment  or  trade-off,  whether 
that  is  additional  cost,  extended  schedule,  or  reduced  performance.  The  key  point 
becomes  whether  the  investment  is  warranted.  This  leads  to  the  question  of  worth,  and 
drives  the  need  to  quantify  the  value  of  flexibility  in  order  to  justify  the  investment. 

We  addressed  the  issue  of  quantifying  the  value  of  flexibility  by  reviewing  the  traditional 
approaches  used  to  make  decisions  under  conditions  of  uncertainty.  We  evaluated 
several  potentially  applicable  techniques;  most  prevalent  being  real  options  analysis. 
However,  the  real  options  approach  has  several  implicit  assumptions  that  may  simply  be 
invalid  or  unverifiable  in  the  DoD  systems  settings.  These  assumptions  could  lead  to 
incorrect  valuation  and  eventually  incorrect  decisions  being  made  about  investment  in 
flexibility.  In  our  analysis,  we  found  that  there  is  little  unifying  theory  or  guidance  on 
best  approaches  to  measure  flexibility,  quantify  value  of  flexibility  in  a  prospective 
systems  acquisition  or  which  MPTs  work  best  in  which  situations. 

Considering  these  major  gaps  in  the  current  state-of-the-art  the  primary  focus  of  our 
research  activities  is  in  developing  a  coherent  value  based  definition  of  flexibility  that  is 
based  on  an  analytical  framework  that  is  mathematically  consistent,  domain 
independent  and  applicable  under  varying  information  levels.  Applying  the  principle  of 
value-driven  design,  which  is  seeks  to  assess  system  value  from  a  more  strategic 
perspective,  thereby  inherently  accounting  for  the  value  of  non-traditional  system 
characteristics  like  flexibility  is  potentially  a  promising  approach  for  valuing  flexibility. 
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The  next  phase  of  this  project  will  focus  on  refining  the  analytical  framework  and 
developing  a  tool  that  can  be  used  by  systems  acquisition  decision-makers  to  conduct 
tradeoff  analysis  between  flexibility  and  other  system  performance  measures  of  interest. 
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Appendices 


Appendix  A:  State-of-the-Art  Survey 


A.1 :  Definitions  and  Measures  of  Flexibility 

The  research  literature  on  flexibility  provides  a  many  different  definitions  for  the  term 
flexibility  and  its  synonyms.  In  order  to  provide  a  clear  definition  and  a  unified 
understanding  of  these  different  terms  the  following  section  provides  an  overview  for 
each  of  the  terms. 

The  first  observation  when  analyzing  the  current  literature  is  the  fact  that  there  does  not 
seem  to  be  a  unified  definition  on  what  flexibility  represents.  Depending  on  the  authors 
the  terms  ’’flexibility”,  “adaptability”,  “agility”  and  ’’reconfigurability”  are  used  as 
synonyms.  Following  the  definition  overview  this  section  also  highlights  the  different 
quantitative  methods  used  to  measure  system  flexibility. 

Rajan  et  al.  [2005]  for  instance  defines  product  flexibility  as  the  degree  of 
responsiveness  for  any  future  change  in  a  product  design.  This  degree  is  based  on  the 
flexibility  of  a  design  for  a  given  change,  the  probability  of  occurrence  and  the  readiness 
of  the  company  to  react  to  this  change.  From  these  factors  the  “Change  Potential 
Number”  (CPN)  is  determined  using  the  change  modes  and  effects  analysis  (CMEA) 
method.  Nilchiani  [2005]  defined  flexibility  as  the  ability  of  a  system  to  adapt  to 
uncertain  internal  or  external  changes  affecting  its  value  delivery,  in  a  timely  and  cost- 
effective  manner.  In  other  words,  flexibility  is  the  ease  with  which  changes  in  value 
delivery  in  a  system  can  be  addressed.  Here  ease  refers  to  the  cost-effectiveness  of 
addressing  change.  The  elements  of  flexibility  are  system  boundary,  time  window  of 
change,  flexibility  aspect,  system  accessibility,  uncertainty,  and  responses  to  change  in 
the  value  delivery  of  the  system.  They  form  the  basis  for  the  measure  of  flexibility 
framework,  which  consists  of  the  problem  definition,  suggested  solution  set,  technical 
design  model,  economic  evaluation  model  (Real  Options  Analysis),  and  finally  the 
flexibility  trade  space  exploration  showing  the  value  gained  through  implementing  an 
alternative  vs.  the  associated  cost. 

McConnell  [2007]  argues  that  while  flexibility  requires  an  outside  agent  as  an  effector 
for  change,  an  adaptable  system  can  change  from  an  internal  effector.  They  go  on  to 
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argue  that  the  difference  between  flexibility  and  adaptability  seems  only  to  reside  in 
where  the  agent  that  effects  the  change  resides  -  external  or  internal  to  the  system. 

An  additional  definition  for  flexibility  is  provided  by  Brown  and  Eremenko  [2008].  They 
state  that  flexibility  is  the  ability  of  a  system  to  change  on  demand.  This  incorporates 
scalability,  evolvability,  maintainability,  and  adaptability.  The  proposed  approach  to 
achieve  flexibility  suggests  fractioning  systems.  The  resulting  fractured  system  provides 
the  ability  to  easily  fulfill  the  above  stated  capabilities.  Its  gain  in  value  (the  total 
lifecycle  value  delivered  by  the  system)  is  compared  against  the  monolithic  approach 
using  Net  Present  Value  (NPV). 

According  to  Bordoloi  et  al.  [1999],  adaptability  is  the  ability  to  change  (e.g.,  to  improve 
performance  over  a  period  of  time)  within  a  given  state.  Their  definition  further 
distinguishing  between  ability  to  change  within  a  state  (adaptability)  and  ability  to 
change  from  one  state  to  another  (flexibility). 

The  following  research  papers  provide  definitions  for  adaptability.  Mark  [2005]  argues 
that  adaptation  is  the  enhancement  or  change  of  a  fielded  system.  If  such  a  change  has  a 
low  cost-benefit  ratio,  as  defined  by  the  customer  or  market,  the  system  is  deemed 
flexible.  Based  on  stakeholder  requirement  Optimal  Point  Designs  (OPD)  are  designed. 
These  OPDs  are  compared  to  Multi-Mission  Capable  designs  (objective  function 
includes  the  functional  requirements  for  each  mission).  Based  on  the  performance  gap, 
captured  by  the  value  of  the  objective  function,  an  “optimal  “  MMC  is  selected.  Gu  et  al. 
[2004],  for  example,  describe  design  adaptability,  and  product  adaptability.  Where 
design  adaptability  refers  to  the  design  paradigm,  which  is  defined  as  the  ability  of  a 
product  to  adapt  to  new  requirements  by  means  of  small  design  changes  that  do  not 
require  a  lot  of  effort.  The  product  adaptability,  on  the  other  hand,  refers  to  a  product 
that  is  adaptable  by  the  user.  Chung  and  Subramanian  [2004]  refer  to  the  adaptability 
of  a  system  as  the  ability  to  accommodate  change  in  its  environment.  Olewnik  et  al. 
[2006]  define  an  adaptable  system  as  one  of  two  modes  of  a  flexible  system.  The  two 
modes  depend  on  whether  the  operating  conditions  or  requirements  change  in  a 
predictable  or  unpredictable  way.  Systems  that  are  able  to  accommodate  an 
unpredictable  change  are  called  robust.  In  case  of  a  predictable  change  to  the  system  it 
is  referred  to  as  adaptable.  Gu  et  al.  makes  a  similar  distinction  of  adaptability  as 
Olewnik,  however,  they  define  the  adaptability  of  a  product/system  with  respect  to 
foreseen  or  unpredictable  change  as  specific  adaptability  and  general  adaptability 
respectively.  Olewnik  et  al.  further  distinguish  an  adaptable  system  as  one  where  the 
change  can  be  either  made  in  real  time  (active)  or  when  the  system  is  not  in  use 
(passive). 

Chmarra  et  al.  [2008]  follows  a  similar  definition  as  Gu  et  al.  with  respect  to  product 
live-cycle  adaptation.  They  distinguish  between  adaptability  during  design  and 
adaptability  when  the  product  performs  a  task,  and  refer  to  these  as  design-time  and 
runtime  adaptability  respectively.  Oppermann  [1994]  makes  a  distinction  between  an 
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adaptive  and  an  adaptable  system.  Where  an  adaptive  system  can  configure  itself 
automatically  according  to  user  needs.  The  adaptable  system,  however,  requires  user 
interaction  to  a  change  in  the  environment. 

Finally,  the  last  synonym  used  in  conjunction  with  flexibility,  reconfigurability,  is 
defined  as  follows.  A  reconfigurable  system,  according  to  Ferguson  and  Lewis  [2006],  is 
one  in  which  variables  can  be  changed  in  real-time  based  on  changes  in  operational 
conditions  or  requirements  in  order  to  maintain  a  high  level  of  performance.  The  Initial 
starting  condition  is  the  design  of  optimized  systems  based  on  each  known  operation 
condition  or  objective.  Based  on  stability  conditions  for  the  system  trajectories  are 
determined  between  the  OPDs.  Finally  a  controller  is  used  for  effective  trajectory 
tracking  (based  upon  a  linear  quadratic  regulator  approach). 

Schulz  et  al.  [2000]  provides  definitions  for  flexibility,  robustness,  adaptability,  and 
agility  in  relation  to  system  intelligence.  According  to  their  definition  flexibility 
represents  a  system  that  can  be  changed  easily  and  without  ill  effect.  Agility  is  defined  as 
the  speed  of  change.  However,  agility  is  also  the  evolutionary  level  of  flexibility.  A 
similar  definition  is  derived  for  robustness,  which  a  system  is  if  it  is  not  affected  by 
changing  environments,  and  adaptability,  characterizing  a  systems  ability  to  adapt  itself 
under  changing  environments. 

The  definition  of  agility  can  be  further  separated.  Haberfellner  et  al.  [2005]  provides  a 
detailed  definition  between  agile  systems  engineering  and  agile  systems.  The  former 
focuses  on  the  alternative  design  space  exploration  during  product  development.  The 
latter  case  describes  the  systems  ability  to  respond  to  changes  in  requirements  after 
initial  fielding. 

Given  the  above  definitions  for  flexibility  and  its  synonyms,  the  following  provides  an 
overview  of  how  flexibility  is  measured  in  the  current  literature.  Based  on  the  previous 
definitions,  flexibility  is  about  the  potential  to  change,  and  thus,  unlike  system 
performance,  it  is  difficult  to  observe  and  measure.  It  is  a  potential  that  is  not 
observable  under  nominal  operating  conditions. 

In  the  context  of  manufacturing,  for  instance,  flexibility  of  production  systems  is 
measured  using  Perti  Nets,  Decision  Theory,  Economic  Consequences,  and  Physical 
Characteristics.  Other  methods  to  quantify  manufacturing  flexibility  such  as  routing, 
operations,  and  loading  incorporate  the  use  of  entropic  measure  (see  Yao  [1985]  and 
Kumar  [1987]).  Methods  to  measure  flexibility  outside  the  manufacturing  realm, 
however,  vary  based  on  the  system  under  consideration.  For  example,  if  the  flexibility 
measure  is  to  be  applied  at  the  early  design  stage,  Cormier  [2008]  proposes  the  use  of  a 
flow  analysis  table  to  rate  the  flexibility  of  designs.  Thurston  [1991],  instead,  develops  a 
formal  Methodology  for  the  Evaluation  of  Design  Alternatives  (MED A).  The  roots  for 
Thurstons  method  can  be  found  in  classical  utility  theory.  Another  approach,  taken  by 
Gu  et  al.  [2004]  and  Mark  [2005],  to  measure  flexibility  is  based  on  some  form  of 
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performance  measure.  In  the  first  paper  the  normalized  savings  of  the  flexible  product 
are  compared  to  those  of  the  dedicated  product  and  in  Mark’s  paper  the  cost  to  realize 
the  change  becomes  the  basis  for  comparison. 

Additional  methods  and  their  respective  shortcomings  in  measuring  system  flexibility 
are  discussed  in  greater  detail  below. 

•  Flow  analysis  tables  characterize  the  interdependency  of  components  and  provide  a 
quantitative  flexibility  metric  for  product  flexibility  with  respect  to  flows  between 
subsystems,  interfaces  between  different  subsystems,  and  the  overall  product  layout. 
However,  it  assumes  different  areas  of  flexibility  are  independent,  and  that  the 
design  space  is  mapped  linearly  to  the  performance  space. 

•  Entropy  measure  is  applied  to  processes  in  the  manufacturing  context  to  determine 
various  dimensions.  Entropy  measure,  however,  is  ill  suited  for  measure  of  flexibility 
since  it  does  not  consider  that  small  number  of  outputs  may  be  more  different  in 
kind  from  each  other  than  a  large  number. 

•  Overlap  Measure  Method  is  a  metric  that  combines  the  uncertainty  range  of  the 
attribute  value  for  a  given  alternative  and  the  decision  maker's  preference  function 
for  the  attribute,  to  determine  a  dimensionless  score.  It  uses  utility  theory  to 
measure  the  requirement  satisfaction.  Its  limitation  lies  within  the  condition  that  it 
can  handle  at  most  one  decision  maker. 

•  The  Generalized  Information  Network  Analysis  (GINA)  compares  multiple  design 
alternatives  on  a  common  basis.  It  uses  information  theory  to  assess  the 
performance  of  systems  during  operation.  However,  the  use  of  information  theory  to 
model  Concept  of  Operations  is  limited  due  to  the  inability  to  capture  physical 
translation. 

•  The  distance  measure  along  the  Pareto  Front  represents  the  set  of  designs  that  are 
non-dominated  across  objectives.  Distance  serves  as  proxy  for  the  'cost'  of  flexibility. 
This  method  only  considers  the  utility  of  the  system  in  a  static  context  and  does  not 
consider  effects  of  context  changes  on  the  system.  Consideration  of  context  change, 
however,  is  essential  to  mitigate  risk  that  comes  from  uncertain  futures.  Also, 
'distance'  as  a  measure  of  flexibility  does  not  capture  the  complex  relationship 
between  individual  design  choices. 

•  Benchmark  Evaluation,  a  method  where  Optimal  Point  Designs  (OPD)  are  evaluated 
against  Multi-Mission  Capable  (MMC)  design  and  Platform  Based  Derivatives  (PBD) 
with  the  OPD  serving  as  an  upper  bound  on  the  MMC/PBD,  uses  a  normalized 
performance  metric  to  measure  flexibility  of  platform  designs.  This  measure  does  not 
consider  uncertainties  due  to  non  orthogonal  performance  basis  resulting  from 
multiple  different  platforms  under  uncertainty. 

As  one  might  expect,  the  definition  of  flexibility  is  “context-sensitive”  [  Kumar,  1999] 
and  that  its  usage  is  “likely  to  be  significantly  different  depending  on  the  domain  of 
applicability”  [Saleh,  2009].  Yet  there  appears  to  be  a  field  of  application  where  the 
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concept  of  flexibility  has  been  extensively  analyzed  and  its  usage  might  be  considered 
relatively  mature  [Fitzgerald,  2009;  Saleh,  2003;  Thomke,  1998]. 

Within  the  manufacturing  literature,  the  definitions  of  flexibility  tend  to  be  somewhat 
more  precise  and  more  consistent.  In  fact,  experts  on  manufacturing  principles  are 
comfortable  enough  with  the  umbrella-term  of  “flexibility,”  that  they  have  created  a 
multitude  of  different  types  of  flexibility  to  describe  more  specific  flexibility-related 
parameters.  And  the  definitions  for  each  of  these  flexibility  “subtypes”  also  tend  to  be 
fairly  consistent.  These  flexibility  subtypes  include  mix  flexibility,  volume  flexibility, 
material  flexibility ,  product  flexibility ,  routing  flexibility,  and  operational  flexibility , 
just  to  name  a  few  [Saleh,  2003;  Baykasoglu,  2009;  Nilchiani,  2006;  Ajah,  2005; 
Bordoloi,  1999;  Saleh,  2001;  36  Saleh,  2009;  Chryssolouris,  1996].  And  while  these 
terms  tend  to  be  internally  consistent,  they  do  not  translate  across  domains.  For 
instance,  “product  flexibility”  and  “operational  flexibility”  are  likely  to  connote 
something  very  different  to  the  acquirer  and  the  operator  than  they  do  to  the 
manufacturer  [Saleh,  2009]. 

Of  all  the  specific  manufacturing-specific  flexibility  subtypes,  none  seems  to  be  precisely 
what  we  are  interested  in  exploring,  though  mix  flexibility  may  be  the  most  germane.  It 
is  defined  as  “the  ability  to  manufacture  a  variety  of  products  without  major 
modification  of  existing  facilities”  [Saleh,  2003].  The  basic  notion  of  easily  expanding 
capability  is  what  we  wish  to  achieve,  and  is  central  to  most  of  the  definitions  that 
pertain  to  Design  Flexibility. 

Scouring  the  literature  on  flexibility,  we  find  that  all  remotely  viable  definitions  do  share 
the  common  element  of  change.  However,  not  all  definitions  agree  that  it  must  be  the 
system  that  undergoes  change  in  order  for  it  to  be  deemed  flexible.  Moreover,  the 
nature  of  the  change,  its  source,  when  it  occurs,  and  how  it  occurs  are  all  potentially 
differentiating  elements  of  the  published  definitions. 

We  do  find,  though,  that  the  elements  of  the  flexibility  definitions  can  generally  be 
wholly  described  by  the  answers  to  one  or  more  (usually  more)  of  the  following  five 
questions: 

1.  Will  the  System  Change?  This  refers  to  whether  the  system  under  consideration 
must  change  in  some  manner  to  be  considered  flexible.  The  alternative  is  that 
the  system  does  not  necessarily  change  itself,  but  rather  “accommodates”  the 
instigating  change. 

2.  What  Measure(s)  of  Change  Efficiency  Applies?  This  aspect  of  the  definition 
provides  a  description  of  how  efficient  the  system  change  is,  relative  to  resources 
like  time  and  money. 

3.  What  is  the  Source  of  the  Change?  Asking  this  tells  us  where  the  instigating 
change  force  is  relative  to  the  system,  i.e.,  internal  or  external. 
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4.  Is  the  Change  Foreseeable?  Aims  to  capture  whether  the  potential  change  is  one 
that  can  be  anticipated. 

5.  When  may  the  Change  Occur?  This  question  relates  to  when  in  the  system’s 
lifecycle  it  may  be  exposed  to  the  change,  with  the  delineation  being  whether  the 
change  occurs  before  or  after  the  system  is  fielded. 

By  way  of  example,  let’s  use  this  construct  to  parse  one  of  the  most  comprehensive 
definitions  in  the  literature.  In  discussing  design  flexibility,  [Saleh,  2009]  states  it  “will 
allow  the  system  to  be  easily  modified  should  these  requirements  change  after  it  has 
been  fielded,”  and  that  “The  requirement  changes  can  be  known  or  unknown  upfront” 
(emphasis  added).  Posing  the  five  questions  above  to  Saleh’s  definition— 

1.  Will  the  System  Change?  Yes.  The  product  has  the  capability  to  be  “modified.” 

2.  What  Measure(s)  of  Change  Efficiency  Applies?  The  efficiency  of  the  change  for 
this  definition  of  flexibility  is  qualified  only  in  terms  of  “easily”;  some  of  the 
definitions  below  will  provide  greater  specificity  for  this  parameter. 

3.  What  is  the  Source  of  the  Change?  “Requirements”  are  the  source  of  change. 

4.  Is  the  Change  Foreseeable?  The  change  may  be  anticipated  (i.e.,  “known”)  or 
unanticipated  (i.e.,  “unknown”). 

5.  When  May  the  Change  Occur?  The  change  would  need  to  occur  during  the 
system’s  operational  phase,  i.e.,  “after  it  has  been  fielded” 

Applying  this  framework  to  all  definitions  encountered  which  were  at  least  marginally 
useful  yields  the  following  general  results,  and  can  be  viewed  in  aggregate  in  Table  1. 

Will  the  System  Change?  Among  those  scholars  who  believe  that  a  system  change  is 
integral  to  the  definition  of  flexibility,  some  are  relatively  specific,  describing  the  system 
as  being  “modified”  [Lafleur,  2010;  Saleh,  2009;  Thomke,  1998],  “redesigned”  [Rajan, 
2003;  Keese,  2007],  or  “reconfigured”  [Olewnik,  2001];  others  are  more  vague,  referring 
to  changing  performance  [Roser,  1999],  the  ability  to  “adapt”  [Fitzgerald,  2009; 
Nachtwey,  2009;  Merriam-Webster  Online  Dictionary  2010],  or  simply  the  ability  to 
change  [Brown,  2009;  Shah,  2008;  Bordoloi,  1999;  Schulz,  1999.  Again,  it  should  be 
noted  that  when  authors  choose  to  define  flexibility  in  terms  of  “adapt”  or  “adaptability,” 
the  result  is  rarely  elucidating. 

Other  authors,  however,  do  not  explicitly  state  that  a  system  must  undergo  a  change  in 
order  to  be  considered  flexible.  One  of  these  authors  makes  no  reference  at  all  to  system 
change  [Viscito, Lauren  2009],  while  the  remainder  seem  to  be  indicating  that  for  a 
system  to  be  considered  flexible,  it  merely  needs  to  accommodate  external  changes.  For 
example,  these  sources  may  refer  to  the  ability  to  “support  new  functions”  [Banerjee, 
2004],  “accommodate  new  requirements”  [Sivanthi,  2008],  or  “to  respond”  [Nilchiani, 
2006;  Qureshi,  2006].  Interestingly,  the  fact  that  this  could  be  a  potential  point  of 
confusion  is  not  addressed  explicitly  in  the  literature,  but  this  is  a  crucial  distinction.  In 
practice,  the  question  of  whether  the  system  must  undergo  change  in  order  to  be 
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considered  flexible  is  essentially  the  difference  between  overcapacitizing  the  system 
from  the  outset  and  “scarring”  it  for  expansion  (see  versatility  discussion  at  the  end  of 
this  chapter).  Using  the  analogy  of  computers,  it  is  the  choice  between  initially 
providing  more  memory  than  required  and  providing  an  expansion  slot  for  additional 
memory  should  it  be  required  later.  If,  in  fact,  both  approaches  may  be  considered 
flexible,  then  we  need  clarifying  terminology  regarding  the  type  of  flexibility  to  which  we 
are  referring  as  the  two  approaches  have  markedly  different  implications  for  system 
design,  development,  operations,  and  maintenance.  While  overcapacitizing  is  one  way 
to  view  this  approach  to  achieving  flexibility,  another  way  to  view  it  is  to  see  it  as 
providing  appropriate  capacity  for  a  broader  range  of  applications.  Continuing  the 
computer  analogy,  we  could  state  that  an  Apple  iPad  is  more  flexible  (perhaps  more 
correctly  stated  as  more  versatile)  than  the  Amazon  Kindle  e-Reader  because  it  has  a  full 
featured  web  browser  and  media  player  included  in  the  device.  A  laptop  computer  may 
provide  even  greater  flexibility  (or  versatility)  than  the  iPad  by  virtue  of  its  ability  to  run 
additional  software  and  support  external  devices  (although  one  could  argue  that  each  of 
these  provides  an  increasingly  degraded  reader  function).  This  discussion  of  versatility 
and  its  relationship  to  flexibility  will  be  discussed  further  at  the  end  of  this  section. 

What  Measure(s)  of  Change  Efficiency  Applies?  A  majority  of  the  useful  definitions  in 
the  literature  qualify  the  definition  of  flexibility  in  terms  of  its  efficiency.  Some 
qualifiers  are  generic,  such  as  change  with  “ease”  or  “easily”  [Lafleur,  2010;  Schulz,. 
1999;  Saleh,  2009;  Rajan,  2003],  “effectively”  [Nachtwey,  2009],  “degree  of 
responsiveness”  [Qureshi,  2006],  or  simply  “ready”  [Merriam-Webster  Online 
Dictionary  2010].  Several  other  definitions  are  more  specific  using  temporal  qualifiers 
like  “timely”  [Saleh,  2009;  Nilchiani,  2006],  “quickly”  [Keese,  2007]  or  “real-time” 
[Olewnik,  2001]  and  monetary  qualifiers  like  “cost-effective”  [Nilchiani,  2006;  [Saleh, [ 
2009;  Keese,  2008;  Sivanthi,  2008]  and  “inexpensively”  [Keese,  2007].  A  few  others 
just  combine  the  two  concepts  into  a  single  phrase  like  “minor  time  and  costs”  [Roser, 
1999],  and  “incremental  cost  and  time”  [Thomke,  1998]. 

It  can  be  argued  that  definitions  that  lack  any  qualification  of  how  easily  the  change  can 
be  effected  are  of  dubious  value.  For  instance,  when  Fitzgerald  [2009]  says  that 
flexibility  is  the  “ability  to  adapt  to  new  circumstances,”  one  wonders  what  system  does 
not  have  the  ability  to  adapt  given  enough  time  and  money.  Under  this  conception, 
every  living  organism  is  flexible  merely  because  it  exists,  and  thus  has  successfully 
adapted  to  new  circumstances.  But  without  any  way  to  discriminate  the  degree  of 
flexibility  among  organisms,  the  definition  can  be  of  no  analytical  value.  Providing 
some  indication  what  we  must  invest  in  order  to  implement  the  system  change  is  critical 
if  we  are  to  have  any  hope  of  quantifying  the  value  of  flexibility. 

What  is  the  Source  of  the  Change?  Just  over  half  of  the  flexibility  definitions  indicate 
what  source,  or  sources,  may  instigate  the  change  in  the  system.  The  basic  distinction  is 
between  “internal”  or  “external.”  Most  authors  do  not  explicitly  state  what  they  mean  by 
internal  and  external,  but  from  those  who  do  in  combination  with  other  contextual 
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clues,  it  is  evident  that  an  internal  change  is  one  that  is  inside  the  system  boundary  such 
as  different  operating  modes,  or  the  deliberate  implementation  of  new  designs. 
Similarly,  external  change  appears  to  refer  to  those  changes  that  occur  outside  the 
system  boundary,  but  still  affect  the  system,  with  the  most  obvious  example  being  the 
operating  environment.  Several  authors  also  characterize  requirement  changes  as  a 
valid  source  of  change,  but  since  it  could  be  argued  that  requirement  changes  may  be 
driven  either  internally  or  externally,  this  category  of  source  change  is  tracked 
separately. 

Of  the  authors  who  chose  to  formulate  their  definition  by  characterizing  the  source  of 
change  as  external  or  internal,  all  agree  that  flexible  systems  must  at  least  accommodate 
externally-induced  changes  [Lafleur,  2010;  Nilchiani,  2003;  Nilchiani,  2006;  Olewnik, 
2001;  Nachtwey,  2009;  Viscito,  2009;  Shah,  2008;  Thomke,  1998;  Ross,  2008].  Two 
sources  also  allow  for  the  source  of  change  to  be  internal  [Thomke,  1998;  Nilchiani, 
2006].  Finally,  a  number  of  others  specifically  call  out  changing  requirements  as  a  valid 
source  of  change  [Lafleur,  2010;  Olewnik,  2001;  Merriam-Webster  Online  Dictionary 
2010;  Keese,  2007;  Saleh,  2009;  Sivanthi,  2008],  with  all  but  one  making  no  comment 
regarding  the  question  of  internal  or  external. 

The  source  of  the  change  is  presumably  linked  to  the  type  of  change,  which  is  ultimately 
what  we  are  interested  in.  The  types  of  changes  that  are  likely  to  be  induced  externally 
are  bound  to  be  different— and  most  likely  more  numerous  and  more  significant— than 
those  that  are  induced  from  within  the  system.  This  aspect  of  the  flexibility  definition  is 
important,  as  characterizing  the  types  and  sources  of  change  that  must  be  accounted  for 
will  necessarily  help  shape  the  techniques  that  can  be  used  to  make  a  system  flexible. 

Is  the  Change  Foreseeable?  Some  experts  felt  it  necessary  to  qualify  their  definition  in 
terms  of  whether  the  change  was  anticipated  or  not.  Of  those  that  did,  all  five  indicated 
that  the  source  of  change  may  be  unknown  [Fitzgerald,  2009;  Nilchiani,  2003;  Olewnik, 
2001;  Keese,  2007;  Saleh,  2009].  Three  of  these  five  also  felt  that  the  source  of  change 
for  a  flexible  system  may  be  known  as  well  [Fitzgerald,  2009;  Olewnik,  2001;  Saleh, 
2009]. 

Whether  a  change  can  reasonably  be  anticipated  is  clearly  important  to  the  system 
designer,  as  “known  unknowns”  can  more  readily  be  characterized  and  mitigated 
through  risk  reduction  techniques.  “Unknown  unknowns,”  however,  are  axiomatically 
more  difficult  to  prepare  for  and  the  designer  may  have  to  revert  to  less  precise 
mitigation  techniques  grounded  in  heuristics  and  “best  practices.” 

When  May  the  Change  Occur?  The  final  definitional  element  that  provides  a 
discriminator  between  the  various  flexibility  definitions  pertains  to  when  the  change 
may  occur.  The  large  majority  of  authors  do  not  address  this  point,  either  apparently 
not  regarding  it  as  relevant,  or  tacitly  accepting  that  the  change  may  occur  at  any  point 
in  the  system  lifecycle.  One  author,  however,  feels  that  a  flexible  system  need  only 
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respond  to  changes  that  occur  after  the  system  is  fielded  [Lafleur,  2010].  Here,  Saleh’s 
careful  distinction  between  “Process  Flexibility”  and  “Design  Flexibility”  leads  to  the 
conclusion  that  process  flexibility  only  applies  to  changes  occurring  “during  the  design 
process,”  and  design  flexibility  applies  to  the  work  done  during  development  that  allows 
“the  system  to  be  easily  modified  should  these  requirements  change  after  it  has  been 
fielded”  [Saleh,  2009]. 

A  summary  of  all  the  definitions  binned  according  to  the  five-question  template  is 
shown  in  Table  1. 

Adaptability  (and  its  Relationship  to  Flexibility) 

Since  our  primary  interest  in  this  report  is  to  study  the  concept  of  flexibility,  the 
research  on  adaptability  is  limited  to  where  adaptability  is  discussed  in  the  context  of— 
or  in  relationship  to— flexibility.  Thus,  when  searching  for  definitions  of  flexibility,  the 
results  were  comprehensive;  whereas  when  searching  for  definitions  of  adaptability,  the 
results  are  constrained  by  the  fact  that  both  concepts  must  be  addressed  in  the  same 
work  (this  same  caveat  applies  to  all  subsequent  terminology  as  well,  but  will  not  be 
repeated).  Nevertheless,  there  were  still  a  decent  number  of  sources  that  sought  to 
define  adaptability.  And  as  we  found  in  the  case  of  flexibility,  all  of  the  viable  definitions 
involve  the  concept  of  change.  And  while  “adaptability”  was  often  used  interchangeably 
with  “flexibility,”  the  overall  variance  within  the  different  definitions  of  adaptability  was 
lower  that  it  was  for  flexibility. 

A  review  of  the  adaptability  material  quickly  shows  that  the  definitions  are  very  similar 
to  those  of  flexibility.  Shah  [2008]  refers  to  the  “subtle  distinction  between  flexibility 
and  adaptability”  contributing  to  the  overall  confusion  on  terminology.  Given  the 
similarity,  we  can  use  the  same  framework  we  used  previously  to  organize  and  analyze 
the  flexibility  definitions.  Of  note,  question  two  is  no  longer  applicable,  and  the 
parameters  have  changed  slightly  for  question  five. 

1.  Will  the  System  Change?  This  refers  to  whether  the  system  under  consideration 
must  change  in  some  manner  to  be  considered  adaptable.  The  alternative  is  that 
the  system  does  not  necessarily  change,  per  se,  but  rather  “accommodates”  the 
change. 

2.  What  Measure(s)  of  Change  Efficiency  Applies?  This  question  is  not  applicable  to 
adaptability— strangely,  none  of  the  definitions  qualified  the  measure  of 
adaptability  in  any  efficiency  terms. 

3.  What  is  the  Source  of  the  Change?  This  question  asks  where  the  instigating 
change  force  is  relative  to  the  system,  i.e.,  internal  or  external. 

4.  Is  the  Change  Foreseeable?  Aims  to  capture  whether  the  potential  change  is  one 
that  can  be  anticipated. 

5.  When  May  the  Change  Occur?  For  adaptability,  this  question  refers  to  a  different 
parameter  than  it  did  for  flexibility.  For  flexibility,  we  were  interested  in  the 
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point  in  a  system’s  lifecycle  where  the  change  could  be  introduced.  For 
adaptability,  the  question  of  “when”  now  refers  to  whether  the  system  can  effect 
the  change  in  real-time,  or  must  be  taken  offline. 

Will  the  System  Change?  All  but  one  source  indicated  that  a  required  element  of 
adaptability  is  for  the  system  to  change  in  some  manner.  As  was  the  case  with 
flexibility,  the  actual  description  of  change  varied  in  terms  of  its  specificity,  and  included 
terms  like  “reconfigure”  [Brown,  2008],  “alter”  [Ross,  2008],  “react”  [Haubelt,  2002], 
and  even  the  redundant  “adapt  itself’  [Schulz,  1999].  The  lone  exception  was  a 
definition  that  refers  to  “accommodating  change”  [Chung,  2004].  Based  on  just  this 
element  of  our  definitional  construct,  there  is  no  meaningful  difference  between 
flexibility  and  adaptability. 

What  Measurelsl  of  Change  Efficiency  Applies?  Not  applicable. 

What  is  the  Source  of  the  Change?  All  but  one  source  felt  that  the  source  of  the  change 
was  relevant  to  the  principle  of  adaptability.  Unfortunately,  the  sources  don’t  agree  on 
the  source.  Four  papers  indicated  that  adaptability  implies  that  the  source  of  change  is 
external  [Olewnik,  2001;  Schulz,  1999;  Haubelt,  2002;  Chung,  2004],  while  three  other 
sources  stated  that  changes  related  to  adaptability  must  be  internal  [Ross,  2008;  Shah, 
2008;  Brown,  2008].  Meanwhile,  just  one  author  made  any  reference  to  changes  in 
requirements  [Olewnik,  2001],  whose  focus  was  the  software  domain. 

Even  with  the  lack  of  consistency  regarding  this  aspect  of  the  definition  of  adaptability, 
we  can  discern  some  difference  from  flexibility.  First,  for  flexibility,  the  source  of 
change  was  overwhelmingly  seen  as  being  external,  with  only  two  of  the  nine  authors 
who  referenced  the  source  of  change  indicating  that  an  internal  source  could  be  a  valid 
source  of  change  for  a  system  to  be  flexible,  and  in  both  cases,  this  was  in  addition  to  an 
external  source.  For  adaptability,  on  the  other  hand,  three  of  seven  authors  went  with 
internal,  and  none  said  it  could  be  either  internal  or  external.  Of  note,  two  authors 
specifically  used  this  criterion  as  a  discriminator  between  flexibility  and  adaptability. 
Ross  [2008]  posits,  “If  the  change  agent  is  external  to  the  system,  then  the  change  under 
consideration  is  a  flexible-type  change.  If  the  change  agent  is  internal  to  the  system, 
then  the  change  under  consideration  is  an  adaptable-type  change.”  Shah  [2008] 
concurs,  arguing,  “Adaptation  is  an  internally  initiated  change,  while  flexibility  is 
externally  initiated.”  Taken  together,  there  appears  to  be  a  nuance  that  adaptability  is 
slightly  more  suggestive  that  the  source  of  change  is  internal  to  the  system.  Here,  Shah 
may  provide  additional  clarity:  “An  adaptable  system  needs  to  incorporate  the  ability  to 
decide  to  instigate  a  change,  while  in  a  flexible  system  the  decision-making  ability  is 
exogenous  to  the  system”  [Shah,  2008]. 

Is  the  Change  Foreseeable?  For  the  definition  of  flexibility,  many  authors  were 
concerned  with  the  notion  of  whether  the  change  could  be  anticipated.  In  the  case  of 
adaptability,  this  factor  appears  to  be  of  little  interest.  Only  one  author  mentioned  this 
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aspect  of  the  definition,  citing  the  applicability  of  “predictable  situations”  [Olewnik, 
2001].  This  may  be  another  potential  discriminator  between  adaptability  and  flexibility. 

When  May  the  Change  Occur?  With  the  new  adaptability  parameters,  this  question 
refers  to  whether  the  system  can  effect  its  change  while  in  an  active  state  (i.e.,  in 
operations)  or  whether  it  must  be  in  a  passive  state  (i.e.,  off-line).  Only  a  couple  of 
authors  included  this  criterion  in  their  definition,  and  their  assertions  were  not  quite  in 
accord.  Olewnik  [2001]  allows  for  the  change  to  occur  while  in  operations  or  off-line, 
while  Haubelt  [2002]  argues  that  the  change  may  only  occur  when  the  system  is  in 
operations.  Of  greater  interest,  however,  is  the  tacit  assumption  among  these  and  other 
authors  that  adaptability  is  only  applicable  to  the  operational  phase  of  a  system.  While 
they  neglected  to  explicitly  include  this  assumption  in  their  formal  definitions, 
contextual  clues  indicate  that  they  felt  adaptability  was  a  concept  that  applied  to 
operations  rather  than  development  [Schulz,  1999;  Brown,  2008]. 

Based  on  the  definitions  found  in  the  relevant  academic  literature,  adaptability  is  a  very 
similar  concept  to  flexibility.  Like  flexibility,  it  refers  to  the  ability  of  a  system  to  change 
itself  when  confronted  by  a  source  of  change.  However,  adaptability  is  distinct  in  that  it 
connotes  that  the  source  of  that  change  is  internal  vie  external,  and  that  the  change 
should  be  encountered  after  the  system  enters  operations,  especially  if  the  change  can  be 
effected  while  the  system  remains  “online.”  Another  distinction  is  that  the  concern  over 
whether  the  change  is  foreseeable  seems  much  less  important  when  discussing 
adaptability.  Finally,  only  flexibility  is  defined  in  terms  of  effecting  the  change  by  some 
measure  of  efficiency,  though  it’s  not  apparent  why  this  should  be. 

Table  2  provides  the  graphical  summary  of  all  of  the  adaptability  definitions  we  have 
examined. 

Robustness  (and  its  Relationship  to  Flexibility) 

While  adaptability  may  be  the  term  most  often  confused  with  flexibility,  it  is  certainly 
not  the  only  one.  The  relationship  between  flexibility  and  robustness  can  be  muddled  as 
well.  One  scholar  asserts  that  “robustness”  and  “adaptability”  are  the  two  design 
components  that  enable  a  system  to  have  flexible  performance  [Olewnik,  2001]. 
Another  scholar  asserts  that  robustness  involves  satisfying  a  fixed  set  of  requirements, 
while  flexibility  is  about  satisfying  changing  requirements  [Saleh,  2003].  Clearly,  we 
need  to  clarify  the  meaning  of  robustness  as  it  pertains  to  flexibility. 

We  again  employ  the  same  framework  we  used  previously  to  organize  and  analyze  the 
flexibility  and  adaptability  definitions.  Of  note,  the  first  question  now  focuses  on  the 
performance  of  the  system,  rather  than  its  inherent  characteristics,  and  the  second 
question  is  again  not  applicable. 
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1.  Will  the  System  Change?  This  refers  to  whether  the  system  under  consideration 
will  undergo  a  change  in  performance,  with  change  being  an  undesired  outcome. 

2.  What  Measurefsl  of  Change  Efficiency  Applies?  This  question  is  not  applicable  to 
robustness— none  of  the  definitions  qualified  the  robustness  in  any  efficiency 
terms. 

3.  What  is  the  Source  of  the  Change?  This  question  asks  where  the  instigating 
change  force  is  relative  to  the  system,  i.e.,  internal  or  external. 

4.  Is  the  Change  Foreseeable?  Aims  to  capture  whether  the  potential  change  is  one 
that  can  be  anticipated. 

5.  When  May  the  Change  Occur?  As  it  did  for  flexibility,  this  question  of  “when” 
refers  to  when  in  the  system’s  lifecycle  it  may  be  exposed  to  the  change,  with  the 
delineation  being  whether  the  change  can  occur  before  or  after  the  system  is 
fielded. 

Will  the  System  Change?  Every  source  agrees  that  for  a  system  to  be  considered  robust, 
it  must  sustain  its  performance  when  exposed  to  change.  Some  definitions  are  more 
explicit  that  the  performance  of  a  robust  system  may  not  change  at  all,  asserting,  for 
example,  that  “robustness  involves  satisfying  a  fixed  set  of  requirements”  [Saleh,  2003] 
and  “robust  systems  deliver  their  intended  functionality”  [Schulz,  1999].  The  phrasing 
in  other  definitions,  on  the  other  hand,  does  seem  to  allow  for  some  leeway  in  system 
performance,  e.g.,  “the  ability  of  the  system  to  continue  delivering  value”  [Shah,  2008] 
or  needing  to  “minimize  the  effect ...  on  the  performance  of  the  system”  [Phadke,  1989]. 
But  whether  system  performance  is  truly  fixed  or  is  allowed  to  slightly  degrade,  it’s  plain 
that  the  salient  characteristic  of  a  robust  system  is  that  it  must  remain  relatively 
constant  when  exposed  to  change. 

What  Measurefsl  of  Change  Efficiency  Applies?  Not  applicable. 

What  is  the  Source  of  the  Change?  The  source  of  the  change  also  appears  to  be  a  key 
component  in  the  definition  of  robustness.  With  one  exception,  every  definition 
indicated  that  an  externally-derived  change  is  a  type  of  change  that  must  be  withstood, 
most  often  referring  to  the  “environment”  [Phadke,  1989;  Olewnik,  2006;  Carlson, 
2002;  Shah,  2008;  Saleh,  2009].  As  summarized  by  Ross  [2008],  “the  more 
environments  to  which  it  is  insensitive,  the  more  robust  the  system.”  Marking  a  small 
departure  from  the  consistency  in  the  robust  definitions  thus  far,  approximately  half  of 
the  definitions  also  allow  for  the  source  of  change  to  be  internally  driven. 

Is  the  Change  Foreseeable?  This  appears  to  be  another  important  parameter  as  nearly 
half  of  the  sources  felt  that  it  was  an  important  part  of  the  definition,  all  of  them 
agreeing  that  the  type  of  change  is  one  that  cannot  be  reasonably  foreseen. 

When  May  the  Change  Occur?  One-third  of  the  sources  considered  here  indicated  that 
robustness  only  applies  to  a  system  once  it  has  entered  operations  [Phadke,  1989; 
Olewnik,  2001;  Schulz,  1999;  Saleh,  2003]. 
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From  the  literature,  there  does  not  appear  too  much  in  the  way  of  a  direct  relationship 
between  flexibility  and  robustness,  though  some  authors  have  identified  some  linkage. 
Olewnik  [2001]  sees  robustness,  along  with  adaptability,  as  “modes  of  flexibility: 
"Adaptable  systems  are  capable  of  accommodating  predictable  changes  in  operating 
environment,  while  robust  systems  are  capable  of  accommodating  unforeseeable 
changes  in  the  operating  environment."  Meanwhile,  Schulz  [1999]  views  robustness  as  a 
prerequisite  to  achieve  adaptability,  i.e.,  “Adaptability  is  the  evolutionary  level  of 
robustness.”  And  Saleh  [2001]  notes  the  similarity  between  the  concepts  but  is  careful 
to  articulate  the  differences  as  well: 

"Although  these  two  concepts  [flexibility  and  robustness]  refer  to  the  ability  of  a  system 
to  handle  change ,  the  nature  of  the  change,  as  well  as  the  system’s  reaction  to  the 
change,  in  each  case  is  very  different:  Flexibility,  as  defined  herein,  implies  the  ability 
of  a  design  to  satisfy  changing  requirements  after  the  system  has  been  fielded,  whereas 
robustness  involves  satisfying  a  fixed  set  of  requirements  despite  changes  in  the 
system’s  environment  or  within  the  system  itself.” 

Another  author  establishes  the  distinction  based  on  the  stability  of  system  objectives 
and  the  nature  of  the  operating  environment.  For  an  entirely  straightforward  design 
task  involving  fixed  objectives  in  an  unchanging  environment,  then  the  system  principle 
is  optimization.  If  there  is  uncertainty  with  respect  to  the  environment,  but  the 
requirements  are  still  fixed,  then  robustness  is  needed.  If  both  the  system  objectives 
and  the  operating  environments  are  uncertain,  then  flexibility  is  what  is  called  for 
[Banerjee,  2004].  Saleh  [2001]  summarizes  the  difference  between  flexibility  and 
robustness  through  the  example  of  a  satellite  system.  If  our  goal  were  to  maintain 
existing,  on-board  functionalities  “despite  changes  in  software  and  hardware 
characteristics  due  to  radiation  impacts,  malfunctions,  aging,  etc,.”  the  system  would 
need  to  have  a  robust  design.  If  instead,  we  are  to  create  “new  functionalities  on-board 
for  changes  in  requirements  occurring  after  launch,  as  events  unfold,  new  environments 
are  explored,  and/or  new  data  becomes  available,  etc”  then  we  must  have  a  flexible 
design. 

Although  the  term,  “robustness,”  was  occasionally  used  a  bit  carelessly  in  the  literature, 
when  it  was  defined,  the  results  were  remarkably  consistent.  The  one  definitional 
outlier  came  from  Dr.  Sambur,  the  2004  SecAF,  who  described  robustness  quite 
broadly,  using  terms  such  as  adaptable,  expandable,  scalable,  reliable,  affordable,  and 
modifiable  [Ross,  2008].  This  definition  notwithstanding,  it  seems  implausible  that 
when  some  scholars  use  the  term  “robustness”  as  a  proxy  for  “flexibility,”  they  are 
misusing  the  former  term.  Rather,  it  appears  more  likely  that  it  is  just  another  example 
of  imprecise  usage  of  “flexibility,”  as  the  difference  between  the  two  terms  is  stark. 
Whereas  flexibility  (and  adaptability)  is  about  the  system  changing  in  the  face  of  change, 
robustness  is  fundamentally  about  the  system  NOT  changing  in  the  face  of  change. 
Consequently,  it  would  clearly  be  incorrect  to  refer  to  a  system  as  “flexible”  merely 
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because  it  was  capable  of  maintaining  a  level  of  performance  in  light  of  a  stressing 
operational  environment. 

The  summary  of  the  robustness  definitions  is  provided  in  Table  3. 

Agility  (and  its  Relationship  to  Flexibility) 

Before  beginning  this  next  discussion  on  agility,  a  quick  note  on  format.  For  agility,  as 
well  as  all  remaining  terminology  to  be  discussed,  we  abandon  the  five-question 
analytical  framework  that  we  applied  to  flexibility,  adaptability,  and  robustness.  This  is 
because  these  upcoming  terms  are  not  well  suited  to  the  model,  and/or  there  is  not 
enough  material  to  make  the  approach  particularly  elucidating. 

Agility  is  yet  another  term  that  is  sometimes  associated  with  the  concept  of  flexibility. 
From  previous  research,  we  find  that  agility— like  the  other  ilities”  discussed  thus  far— 
revolves  around  change.  In  the  case  of  agile  systems,  the  focus  seems  to  be  on  the  speed 
with  which  the  system  can  effect  the  change: 

•  “The  ability  of  a  system  to  make  a  change  quickly.”  [Ross,  2008] 

•  “The  ability  to  change  rapidly ”  [  Shah,  2008] 

•  “represents  the  property  of  a  system  to  implement  necessary  changes  rapidly” 
[Schulz,  1999] 

Sieger  [2000]  conveys  a  similar  meaning  when  examining  the  agility  of  an  organization, 
but  the  element  of  cost  is  also  incorporated:  “the  ability  of  an  organization  to  respond 
quickly  and  cost  effectively  to  unexpected  changes  in  customer  desire.” 

Unfortunately,  other  definitions  are  virtually  indistinguishable  from  a  definition  of 
flexibility— 

•  "The  ability  of  a  system  to  be  modified  or  adapt  itself  to  wholly  unanticipated 
operating  conditions  or  functional  requirements”  [Banerjee,  2004] 

•  “The  ability  to  respond  with  ease  to  unexpected  but  anticipated  events”  [Oleson, 
1998] 

Oleson’s  definition  adds  the  additional  challenge  of  forcing  the  reader  to  ponder  how  an 
event  can  be  both  “unexpected”  and  “anticipated.”  And  the  final  definition  views  agility 
in  a  completely  different  light: 

•  “The  ability  to  instigate  change  rather  than  react  to  it.”  [Upton,  1994] 

Still  others  formulate  definitions  of  agility  that  encompass  adaptability,  flexibility,  and 
robustness  [Dove,  2005].  In  terms  of  the  relationship  between  agility  and  flexibility, 
Schulz  [1999]  regards  flexibility  as  “a  prerequisite  to  achieve  agility,  i.e.,  agility  is  the 
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evolutionary  level  of  flexibility.”  Ross  [2008]  discriminates  the  concepts  as  follows: 
“Agility  is  a  modifier  describing  the  nature  of  the  change,  just  as  flexibility  and 
adaptability  describe  the  location  of  the  change  agent.” 

In  general,  agility  is  encountered  much  less  frequently  in  the  published  literature  than 
flexibility,  but  the  terms  are  clearly  closely  related.  Both  concepts  refer  to  the  ability  of 
the  system  to  change,  and  both  terms  provide  qualifications  related  to  the  source  and 
nature  of  the  change  to  be  responded  to.  And  in  one  respect  agility  is  more  like 
flexibility  than  adaptability  is.  Unlike  adaptability,  agility  does  include  the  element  of 
efficiency,  particularly  with  respect  to  the  speed  of  change.  For  these  reasons,  it  appears 
that  agility  may  best  be  regarded  as  a  clarifying  component  of  flexibility,  as  in  a  flexible 
system  capable  of  implementing  a  change  very  quickly  (and  perhaps  inexpensively) 
would  be  considered  highly  agile. 

Changeability  (and  its  Relationship  to  Flexibility  and  Other  “-ilities”) 

As  if  the  existing  change-related  terminology  was  not  convoluted  enough,  it  turns  out 
that  an  increasing  number  of  authors  have  begun  to  discuss  “changeability”  as  an 
umbrella  term  for  the  concept  of  system  change.  Nachtwey  [2009]  regards  flexibility  as 
a  “subset  of  changeability,”  while  Fricke  [2005]  and  Shah  [2008]  argue  that  the  four 
terms  we  have  discussed  thus  far  are  the  very  same  that  comprise  changeability: 
“Changeability  is  defined  as  the  ability  of  a  system  to  change  easily,  and  can  be 
decomposed  into  four  categories:  robustness,  agility,  adaptability,  and  flexibility.” 

Ross  [2008]  largely  agree  but  replace  “agility”  with  the  terms  “scalability”  and 
“modifiability,”  and  then  expound  the  notion  of  changeability  considerably.  Starting 
with  the  basic  idea  that  the  “Changeability  of  a  system  is  determined  by  how  easily  it  can 
undergo  various  changes,”  they  develop  a  framework  that  consists  of  three  fundamental 
aspects  of  change:  change  agents  (instigator,  or  force,  for  the  change),  change 
mechanisms  (the  path  taken  in  order  to  reach  state  2  from  state  1),  and  change  effects 
(actual  difference  between  the  origin  and  destination  states).  Tying  this  approach  back 
into  our  earlier  delineations  of  flexibility  and  adaptability,  Ross,  et  al  describe  change 
agents  that  are  external  to  the  system  (e.g.,  users,  technicians)  as  flexible-type  changes, 
whereas  change  agents  that  are  internal  to  the  system  (e.g.,  automatic  software 
updating)  are  adaptable-type  changes.  Meanwhile,  scalability,  modifiability,  and 
robustness  are  each  a  category  of  change  effect,  “which  are  quantified  differences  in 
system  parameters  before  and  after  a  change  has  occurred.” 

Another  aspect  of  the  changeability  research  is  the  principle  of  how  to  “Design  for 
Changeability,”  or  DFC.  As  described  by  Schulz  [1999],  DFC  also  consists  of  flexibility, 
adaptability,  robustness,  and  agility,  and  together  these  “strategic  attributes  ...  describe 
the  degree  of  intelligence  regarding  the  systems  [sic]  ability  either  to  be  adapted,  or  to 
react  to  changes  itself.” 
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Modularity  (and  its  Relationship  to  Flexibility) 

Modularity  is  undoubtedly  a  viable  research  stream  in  its  own  right.  And  while 
modularity  has  its  own  definitional  challenges,  its  meaning  is  unambiguous  enough  and 
distinct  enough  from  flexibility,  that  there  is  little  chance  of  confusing  the  two  terms. 
Yet,  these  two  non-traditional  system  characteristics  continue  to  be  closely  linked  in  the 
literature,  with  modularity  routinely  described  as  substantially  contributing  to  design 
flexibility,  product  flexibility,  and  overall  flexibility  (e.g.  Rajan,  2003;  Nelson,  1997; 
Gershenson,  2003;  Baldwin,  2000).  If  we  could  establish  that  there  is,  in  fact,  a 
correlation  between  these  two  terms,  then  extant  studies  on  modularity  may  provide 
additional  insights  into  defining,  quantifying,  and  implementing  flexibility. 

We  begin  by  examining  the  definition  of  modularity.  The  IEEE  definition  is  “The  degree 
to  which  a  system  or  computer  program  is  composed  of  discrete  components  such  that  a 
change  to  one  component  has  minimal  impact  on  other  components”  [Institute  of 
Electrical  and  Electronics  Engineers  1990].  This  is  a  good  start  that  manages  to  capture 
a  key  element  of  modularity:  component  independence.  But,  the  full  meaning  of 
modularity  is  richer.  The  best  source  for  a  discussion  on  the  meaning  of  modularity  is 
certainly  Gershenson  [2003],  who  provides  an  outstanding  and  seemingly  exhaustive 
overview  of  existing  research  on  the  definition  of  modular  product  design  and  its 
putative  benefits,  concluding  that  there  are  really  three  central  components  to  the 
principle  of  modularity: 

1.  The  independence  of  a  module’s  components  from  external  components 

2.  The  similarity  of  components  in  a  module  with  respect  to  their  life-cycle 
processes 

3.  The  absence  of  similarities  to  external  components 

Viewed  in  this  manner,  it  is  easy  to  see  why  so  many  authors  regard  modularity  as  an 
essential  enabler  for  achieving  flexibility.  In  theory,  these  modular  principles  should 
limit  the  potential  impact  of  a  change,  while  simultaneously  reducing  the  extent  of 
ramifications  in  the  system’s  response.  More  specifically,  the  independence  of  the 
components  tends  to  isolate  the  impact  of  the  change,  the  similarity  of  components 
tends  to  simplify  the  development  and  implementation  of  a  response  to  the  change  (e.g., 
redesign),  and  the  absence  of  similarities  to  external  components  expands  the  option 
space  and  simplifies  the  change  propagation  analysis.  It  is  the  principle  of 
independence  that  is  most  crucial  in  this  assessment  [Gershenson,  2003]  as  “the  greater 
the  connectivity  between  systems,  the  greater  is  the  chance  that  a  change  to  one  system 
leads  to  changes  in  other  systems”  [Eckert,  2004].  For  all  of  these  reasons,  it  seems 
sensible  to  conclude  that  a  more  modular  system  is  likely  to  provide  greater  flexibility. 

Also  recall  that  a  key  aspect  of  a  flexible  system  is  its  ability  to  respond  to  changes  more 
quickly  and/or  cost-effectively.  These  are  some  of  the  same  benefits  that  compel 
researchers  and  practitioners  to  laud  modularity.  The  idea  that  a  modular  system 
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should  be  able  to  incorporate  changes  more  quickly  or  at  a  lower  cost  is  advocated  by  a 
number  of  studies  that  have  examined  both  modularity  and  flexibility  [Rajan,  2005; 
Ulrich,  1995;  Sanchez,  1996;  Walz,  1980;  Gershenson,  2003;  Thomke,  1997;  Ulrich, 
1999;  Fixson,  2005].  As  Gershenson  [2003]  observes,  “"By  promoting 
interchangeability,  modularity  also  gives  designers  more  flexibility,  with  decreased  cycle 
time,  to  meet  these  changing  processes.”  Rajan  states  flatly,  “design  flexibility  is  directly 
proportional  to  the  number  of  modules  in  the  product”  [Rajan,  2005]. 

Modularity  experts  have  provided  many  other  generalized  reasons  why  modularity 
bolsters  flexibility.  The  list  is  long,  and  includes  assertions  that  modularity— 

•  “Multiplies  design  options  through  mix  and  match  of  modules"  [Baldwin,  2000] 

•  Provides  “manageable  units  of  programs  or  hardware”  [Nelson,  1997] 

•  “Reduces  the  redesign  cost  for  any  future  change”  [Rajan,  2005] 

•  “Helps  to  define  the  interfaces  between  components”  [Stryker,  2009] 

•  Allows  a  system  to  "more  readily  adapt  ...  by  ‘plugging  in’  new  modules” 
[Gershenson,  2003] 

•  “Allows  the  'mixing  and  matching'  of  modular  components  to  give  a  potentially 
large  number  of  product  variations”  [Sanchez,  1996] 

•  “Allows  a  designer  to  control  the  degree  to  which  changes  in  processes  or 
requirements  affect  the  product”  [Gershenson,  2003] 

•  Allows  for  “relatively  few  designs  to  meet  [a]  greater  number  of  applications” 
[Walz,  1980] 

•  Leads  to  systems  that  “have  higher  adaptability  and  consequently  have  better 
survival  rates  under  changing  requirements”  [Lipson,  2001] 

In  a  study  involving  space  systems  like  Hubble,  Mir,  and  the  International  Space 
Station,  and  assessing  responses  to  foreseen  changes,  it  was  found  that,  “Modular 
design  and  separation  of  functionality  are  recognized  as  likely  flexibility-enabling 
characteristics”  [Lafleur,  2010].  Schulz  [1999]  also  advocate  the  premise  that 
modularity  is  an  essential  aspect  of  flexibility.  As  part  of  their  work,  they  provide  a 
detailed  list  of  “extending  principles”  which  are  said  to  comprise  the  concept  of 
flexibility  (and  adaptability,  according  to  the  authors).  These  principles  included— 

•  Autonomy:  Characterized  by  objects,  which  are  capable  of  providing  basic 
functionality  necessary  to  ensure  their  independence  from  the  embedding 
systems 

•  Scalability:  Based  on  elements  independent  from  scale  (fractals),  architectures 
may  be  scaled  upwards  or  downward 

•  De-Centralization:  Based  on  loose  coupling  and  strong  cohesion 

In  terms  of  the  contrasting  of  these  two  system  features,  Olewnik  [2001]  believes  the  key 
discriminator  is  “type  of  adaptability  utilized.”  For  modular  systems  to  achieve  greater 
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performance,  adaptations  must  occur  offline,  whereas  flexible  systems  can  enhance 
their  performance  while  online. 

It  should  be  noted  that  the  multitude  of  cogent  explanations  for  why  modularity  enables 
flexibility  must  be  regarded  as  somewhat  lacking.  Essentially,  the  arguments  are  based 
on  empirics  and  heuristics,  lacking  any  solid  theoretical  justification.  As  Saleh  [2009] 
observes,  the  association  between  modularity  and  flexibility  is  treated  as  “an  intuitive  or 
self-evident  truth,  although  there  is  limited  theoretical  proof.”  Surveying  the  literature, 
it  is  evident  that  our  ability  to  measure  modularity  is  more  mature  than  our  ability  to 
measure  flexibility  [Gershenson,  2003;  Holtta-Otto,  2007;  Mikkola,  2007;  Stryker, 
2009].  For  instance,  Stryker  [2009],  in  the  context  of  their  modularity  study,  suggest 
that  modularity  has  four  quantifiable  elements:  degree  of  coupling,  reusability, 
reconfigurability,  and  extensibility.  While  focusing  on  the  reconfigurability  element, 
they  develop  a  reconfigurability  measure,  and  observe  that  higher  reconfigurability 
measures  and  “minimization  of  pairwise  constraints”  will  necessarily  allow  for  greater 
flexibility.  Studies  like  these  yield  promise  that  if  a  quantitative  linkage  could  be 
established  between  flexibility  and  modularity,  the  opportunity  arises  to  use  modularity 
as  a  proxy  measure  for  flexibility. 

Before  proceeding  to  the  next  “-ility,”  it  should  be  noted  that  there  were  some 
cautionary  voices  regarding  the  limits  of  how  much  flexibility  could  be  achieved  with 
modularity,  and  potential  conflicts  between  the  two.  The  author  who  developed  a  large 
portion  of  the  “changeability”  framework  believes  that  modularity  should  be  considered 
separately  from  these  other  “-ilities,”  as  it  is  better  regarded  as  an  architectural  concept 
used  as  a  means  to  achieving  change,  rather  than  a  change  agent  or  measure  of  change 
response  [Ross,  2008].  Ethiraj  [2004]  readily  acknowledged  that  some  level  of 
modularity  is  conducive  to  flexibility  (he  refers  to  this  as  “adaptive  change”),  but  asserts, 
“excessive  levels  of  modularity  can,  in  the  limit,  stymie  any  possibility  of  adaptive 
change.”  Ross  [2008]  advises  that  while  modularity  can  increase  some  aspects  of 
system  “scalability,”  other  aspects  may  suffer  due  to  the  up-front  investment  costs  of 
implementing  modularity.  Finally,  there  may  be  other  tradeoffs  related  to 
modularization  and  flexibility: 

"There  appears  to  be  a  potential  trade-off  between  the  desire  for  modularity  from  a 
‘business’  standpoint  and  the  desire  for  high  performance  and  efficiency  in  the 
technical  domain  ...  a  more  modular  product  is  likely  to  be  larger,  heavier,  slower,  and 
less  energy  efficient. ”[Holtta-Otto,  2007] 

The  implication  is  that  if  there  are  downsides  to  implementing  modularity,  and 
modularity  and  flexibility  are  indeed  correlating  design  characteristics,  then  these  same 
downsides  are  likely  to  apply  to  flexibility. 

Interoperability  (and  its  Relationship  to  Flexibility) 
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Like  modularity,  interoperability  is  a  mature  research  topic.  Also,  like  modularity, 
measures  have  been  developed  to  quantify  system  interoperability,  so  there  is  again  the 
prospect  of  using  interoperability  as  proxy  measure  of  flexibility.  Unlike  modularity, 
however,  the  link  between  interoperability  and  flexibility  is  tenuous. 

Not  surprisingly,  there  are  a  number  of  different  definitions  for  interoperability.  From 
IEEE,  interoperability  is  defined  as  "the  ability  of  two  or  more  systems  or  components  to 
exchange  information  and  to  use  the  information  that  has  been  exchanged"  [Institute  of 
Electrical  and  Electronics  Engineers  1990].  The  Department  of  Defense  states  that 
interoperability  is  “the  ability  of  systems,  units,  or  forces  to  provide  services  to  and 
accept  services  from  other  systems,  units,  or  forces  and  to  use  the  services  so  exchanged 
to  enable  them  to  operate  effectively  together"  [Department  of  Defense  2001].  Ford 
[2007]  recently  provide  a  comprehensive  review  of  definitions  related  to  interoperability 
and  found  that  the  DoD  definition  was  easily  the  most  prominent. 

With  respect  to  a  potential  relationship  between  interoperability  and  flexibility,  there  is 
extremely  little  to  be  found.  From  the  Schulz  [1999]  study  introduced  in  the  modularity 
section,  one  of  the  extending  principles  of  flexibility  reads,  “Integrability:  characterized 
by  compatibility  and  interoperability  applying  generic,  open,  or  common/consistent 
interfaces.”  Assuming  a  relationship  does  exist  between  flexibility  and  interoperability, 
arguments  could  be  made  both  for  a  positive  and  negative  correlation.  On  the  one  hand, 
the  capability  to  readily  leverage  other  systems  would  seem  to  provide  greater  flexibility 
of  responses.  On  the  other  hand,  the  link  to  the  other  system  represents  another 
possible  path  of  incurring  change,  and  maintaining  the  interface  link  might  effectively 
constrain  flexibility.  Unfortunately,  the  dearth  of  published  materials  on  this  point 
forces  us  to  only  conjecture. 

Miscellaneous  Related  “-ilities” 


We  will  conclude  this  chapter  on  definition  with  a  short  summary  of  some  miscellaneous 
related  terminology: 

Scalability:  Brown  [2008]  describes  scalability  as  the  “ability  to  add  components  or 
capability  to  a  system  throughout  its  lifetime.”  In  this  view,  it  is  akin  to  the  concept  of 
incremental  deployment.  A  similar  conception  is  provided  by  Ross  [2008],  whom  we 
noted  previously  defines  scalability  in  his  discussion  of  changeability.  Ross  defines 
scalability  as  “the  ability  to  change  the  level  of  a  parameter.”  So,  for  example,  if  an 
engine  were  designed  such  that  it  can  have  three,  four,  six,  or  eight  cylinders,  it  would  be 
scalable  along  this  set  of  parameters. 

Modifiability:  Ross  then  goes  on  to  define  modifiability  as  “the  ability  to  change  the 
membership  of  the  parameter  set.”  Extending  the  engine  example,  if  we  can  add  a 
second  set  of  parameters  related  to  weight  (i.e.,  1000,  1500,  1800,  and  2600),  then  the 
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engine  is  deemed  to  be  modifiable  (but  again,  the  specific  weights  indicate  changes  in 
scale). 

Evolvability:  "Ability  to  replace  components  due  to  technology  obsolescence"  [Brown, 
2008].  This  essentially  represents  the  “upgradeability”  of  the  system. 

Universality:  This  concept  refers  to  the  ability  of  a  system  to  have  broad  application, 
i.e.,  to  “be  used  in  a  variety  of  situations  without  change  or  modification”  [Saleh,  2001]. 
Saleh  is  clear  that  this  descriptor  should  not  be  confused  with  flexibility,  which— as  we 
have  noted— mandates  an  element  of  change  for  the  system.  He  exemplifies  the 
distinction  noting  that  a  satellite  that  carries  multiple  payloads  and  performs  multiple 
missions  would  be  universal,  not  flexible.  The  term  “universality”  tends  to  be  applied 
more  often  in  the  software  domain.  The  broader  term  for  this  same  concept  appears  to 
be  “versatility.” 

Versatility:  This  term  is  typically  used  as  a  synonym  of  universality,  denoting  “a  high 
range  of  capabilities’  variety”  [Fitzgerald,  2009].  The  standard  example  of  a  versatile 
system  is  the  Swiss  army  knife  [Baykasoglu,  2009].  Another  example  would  be  a  pen 
capable  of  writing  upside  down  in  multiple  colors.  Most  authors  would  not  characterize 
the  knife  or  the  pen  as  flexible.  Others  probably  would,  though,  given  the  following 
examples  of  putative  flexibility: 

•  “Nike  shoes  provide  flexibility  to  the  customer  in  terms  of  color  choice, 
customized  emblems  (such  as  college  names,  symbols,  and  mascots),  and  choice 
of  sole  designs.”  [Qureshi,  2006] 

•  “removable-bit  screwdriver”  and  “modern  adjustable  chair”  [Rajan,  2005] 

•  A  pen  with  “three  modes  of  writing  -  black  ink,  red  ink  and  pencil”  [Rajan,  2003] 

As  highlighted  during  the  flexibility  discussion  on  system  change,  the  difference 
between  a  versatile  system  and  a  flexible  system  is  crucial.  Implementing  capabilities  at 
the  beginning  of  a  program  above  and  beyond  the  stated  requirements  is  fundamentally 
a  different  programmatic  and  design  approach  than  providing  a  system  with  the 
capacity  to  expand  its  capabilities  beyond  the  initial  requirements  at  a  later  time.  And 
while  the  user  would  undoubtedly  prefer  the  versatile  system  to  the  flexible  system 
(because  there  would  be  no  need  to  wait  for  the  capability  to  be  implemented),  this  is 
likely  to  be  non-optimal  for  at  least  two  reasons.  The  first  should  be  obvious.  No  system 
can  be  expected  to  provide  all  capabilities,  so  a  decision  must  be  made  based  on  the 
likelihood  of  needing  the  capability  and  the  cost  of  obtaining  it.  By  providing  additional 
capabilities  not  justified  by  formal  requirements,  the  procurement  entity  is  saying  that  it 
knows  more  about  this  cost-benefit  relationship  than  the  user— essentially  indicating 
that  they  should  own  the  requirements! 

The  more  subtle  reason  is  that  there  is  inevitably  a  cost  associated  with  overcapacitizing 
the  versatile  system.  In  fact,  the  question  of  whether  overcapacitizing  is  a  better  option 
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than  scarring  is  analogous  to  the  question  of  whether  scarring  is  a  better  option  than  not 
scarring,  and  is  at  the  heart  of  a  real  option.  Intuitively,  the  cost  difference  between 
scarring  and  not  scarring  seems  significantly  less  than  the  cost  difference  between 
overcapacitizing  and  scarring.  Whether  this  is  true  in  reality,  and  whether  the  cost 
would  be  warranted  in  certain  situations  must  be  a  central  focus  of  this  research.  Thus, 
while  versatility  might  not  be  a  valid  aspect  of  the  flexibility  model,  perhaps  it  is  an 
appropriate  aspect  of  the  value  model.  Put  in  actuarial  terms,  and  paraphrasing 
Nachtwey’s  interpretation  of  flexibility,  when  is  it  worthwhile  to  purchase  the  flexibility 
or  versatility  insurance  options? 

Before  leaving  the  discussion  of  versatility,  it  may  be  useful  to  discuss  versatility  at  the 
concept  level,  because  versatility  at  the  concept  level  may  not  simply  be  the  result  of 
overcapacitizing  the  system.  A  simple  example  may  serve  to  illustrate  this  point.  One 
approach  to  countering  mobile  theater  ballistic  missiles  is  to  provide  air  defense  systems 
(radars  and  interceptor  missiles)  that  are  capable  of  tracking  and  shooting  down 
incoming  warheads  (this  is  often  referred  to  as  active  defense).  While  this  active 
defense  concept  may  be  effective  in  countering  a  current  generation  of  missile  threats,  it 
may  not  be  a  versatile  concept  in  that  it  may  be  completely  ineffective  against  a  new 
threat  of  mobile  ground  launched  cruise  missiles.  Further,  it  may  not  be  considered 
very  flexible  if  it  is  very  difficult  to  modify  the  radars  and/or  interceptors  to  provide  a 
capability  against  cruise  missiles.  An  alternative  concept  such  as  attack  operations 
(going  after  the  mobile  launchers  before  or  after  missile  launch  with  an  air-to-ground 
system)  may  be  considered  more  versatile  (or  flexible)  if  it  can  readily  be  used  (or  easily 
modified  and  used)  to  counter  a  wide  range  of  mobile  missile  launchers  (ballistic,  cruise, 
SAMs,  etc.).  The  attack  operations  concept  may  be  more  or  less  effective  against  the 
initial  requirement  set  associated  with  the  mobile  ballistic  missile  threat;  that  is  not  the 
point  of  this  discussion.  The  attack  operations  concept  in  this  case  may  be  judged  as 
more  versatile  (or  flexible)  when  considered  against  a  broader  range  of  scenarios. 


A.2:  Approaches  for  Valuing  Flexibility 

Given  the  previous  overview  on  definitions  and  methods  used  to  measure  flexibility  the 
goal  of  this  section  is  to  review  the  current  state  of  the  art  literature  on  valuing 
flexibility.  It  provides  an  overview  of  the  different  methods  developed  in  order  to  enable 
decision  makers  to  select  the  system  which  provides  the  best  design  to  adapt  to  future 
uncertainty. 

The  current  literature  on  valuing  flexibility  provides  several  methods  to  determine  the 
value  obtained  through  system  flexibility.  Peoples  [2004],  for  instance,  proposes  a 
program  valuation  technique  which  is  based  on  real  options  theory  (geometric  Brownian 
motion)  to  calculate  E[NPV].  In  a  similar  way  Gonzalez-Zugasti  et  al.  [1999]  apply  the 
Real  Options  concept  to  model  risks  and  delayed  decision  benefits  under  uncertainty. 
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They  develop  a  quantitative  measure  of  the  value  of  different  family  designs  to  select  the 
most  appropriate  design  off  all  design  alternatives. 

Using  Real  Options  as  a  component  in  their  methodology  to  design  and  analyze 
flexibility  in  large-scale  complex  systems  Silver  et  al.  [2007]  synthesize  Real  Options 
Analysis  along  with  the  concepts  of  Decision  Theory,  Network  Optimization,  and 
Scenario  Planning  into  the  concept  of  Time-Expanding  Decision  Network  (TDNs).  This 
concept  provides  a  framework  to  quantify  the  value  of  system  flexibility.  It  consists  of 
five  principle  steps  that  evaluate  different  set  of  designs  based  and  their  switching  cost 
using  minimum  cost  paths  through  a  network  of  chance  and  decision  nodes  until 
convergence. 

A  two  stage  optimized  design  process  for  flexible  product  platform  components  is 
developed  by  Suh  et  al.  [2007].  They  evaluate  the  best  design  based  on  the  Net  Present 
Value  using  Monte  Carlo  simulation.  Another  framework  to  measure  the  value  of 
flexibility  of  fielded  products  is  developed  by  Mark  [2005].  He  determines  the  optimal 
design  by  evaluating  Optimal  Point  Designs  (OPD)  against  Platform  Based  Derivatives 
(PBD).  The  valuation  is  based  on  the  performance  gap  of  the  OPD  versus  the  PBD. 

The  approach  taken  by  Besharati  et  al  [2006]  differs  from  the  previously  described 
approaches  in  that  it  is  based  on  customer  expected  utility  metric  to  support  the  selected 
product  design.  They  use  a  generalized  purchase  modeling  approach  to  develop  a 
Decision  Support  System  (DSS). 

However,  several  of  the  studied  papers  on  valuing  flexibility  have  restrictive 
assumptions  about  the  system  type  to  ensure  analytical  feasibility.  This  restriction 
comes  at  the  expense  of  the  applicability  of  the  model.  Furthermore,  the  method  for 
selecting  the  underlying  stochastic  process  that  affects  the  value  has  not  been  defined 
clearly.  Additional  methods  for  valuing  flexibility  and  their  respective  shortcomings  are 
discussed  below. 

•  Decision  tree  analysis  provides  a  graphical  representation  of  the  decision  process. 
It  can  generate  insight  even  with  little  hard  data.  Thoroughly  assessing  market 
risks,  however,  is  rendered  harder  in  decision  trees.  This  is  amplified  by  the  fact 
that  the  analysis  most  often  is  based  on  subjective  rather  than  an  objective 
market-based  approach.  The  subjectivity  is  a  result  of  the  assumptions  made 
during  the  cash  flow  estimation. 

•  Real  Options  Analysis  calculates  the  value  of  project  flexibility  in  a  managerial 
context.  It  is  useful  in  its  ability  to  provide  a  financial  value  of  having  options  for 
design  trade-offs.  However,  Real  Options  can  only  access  the  value  for  one 
particular  option.  In  a  setting  with  multiple  options  for  a  single  system  Real 
Options  is  not  able  to  determine  (value)  which  option  to  exercise. 

•  Value  Weighted  Filtered  Outdegree  is  a  metric  to  identify  valuably  flexible 
designs  in  a  tradespace.  It  relies  on  Multi-Attribute  Tradespace  Exploration 
(MATE)  to  evaluate  the  performance  of  many  different  designs  in  utility-cost 
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space.  However,  the  results  of  this  method  are  dependent  on  the  chosen 
transition  rule.  In  addition,  the  metric  is  dependent  on  the  tradespace  sampling 
strategy  used  by  the  designer.  Finally,  the  process  is  time  consuming  (costly) 
because  it  requires  significant  input  from  the  stakeholders  and  domain  expertise 
making  it  less  likely  candidate  for  the  rapid  study  of  system  flexibility. 

In  order  to  overcome  these  shortcomings  and  focus  on  the  identification  and 
quantification  of  changeability  the  work  by  Ross  et  al.  [2008]  develops  a  framework 
that  intentionally  excludes  the  valuation  of  flexibility.  They  apply  tradespace  exploration 
in  order  to  compare  and  determine  which  design  is  more  changeable.  By  doing  so  they 
provide  a  framework  that  does  not  rely  upon  specific  assumptions  regarding  how  to 
collapse  time,  utility,  cost,  and  uncertainty  into  a  single  metric.  In  order  to  value  a 
specific  design  valuation  methods  can  be  applies  on  top  of  the  proposed  framework. 

A  similar  focus  is  found  in  the  work  by  Nilchiani  et  al.  [2007].  They  present  a  six- 
element  framework  for  measuring  space  system  flexibility.  The  ability  of  this  framework 
is  to  value  flexibility  based  on  monetary  or  non-monetary  value  by  relying  on  aspects  of 
interest  for  evaluation.  Baseline  and  alternative  designs  are  evaluated  using  the 
appropriate  evaluation  methodology.  They  distinguish  between  crude  uncertainty 
capture  (Net  Present  Value),  technical  uncertainties  (decision  analysis  techniques),  and 
market  uncertainty  (option  pricing  theory).  For  the  non-monetary  case  they  propose 
decision  tree  analysis  combined  with  utility  theory  or  prospect  theory. 

NPV 

Based  on  the  literature,  we  know  that  the  value  of  flexibility  is  positively  correlated  to 
uncertainty,  such  that  the  greater  the  uncertainty  in  the  system,  the  greater  the  value  a 
flexible  design  option  is  likely  to  have.  So  if  we  are  to  make  any  headway  on  quantifying 
the  value  of  flexibility,  we  need  the  ability  to  make  the  best  decision  under  conditions  of 
uncertainty.  Fortunately  for  us,  this  type  of  problem  has  been  studied  extensively  in 
economics. 

One  approach  is  net  present  value  (NPV)  analysis.  NPV  is  a  standard  method  for 
determining  the  time  value  of  money.  It  takes  into  account  the  net  cash  flow  at  a 
particular  time  t,  as  well  as  the  required  rate  of  return  (also  known  as  the  discount 
rate).  Thus,  the  expected  cash  flows  are  discounted  at  an  interest  rate  that  accounts  for 
the  time  value  of  money  as  well  as  the  project  risk  [Ekstrom,  2005].  Several  studies  use 
NPV  as  part  of  their  effort  to  quantify  flexibility,  including  [Kumar,  1999;  Olewnik, 
2006;  Sivanthi,  2008;  Suh,  2007;  Brown,  2009]. 

For  the  most  part,  though,  researchers  tend  not  to  be  in  favor  of  using  NPV  for  decisions 
involving  flexibility  [Saleh,  2003].  While  NPV  is  sufficient  in  cases  of  “low  uncertainty, 
or  [when]  you  have  no  scope  to  change  course,”  [Copeland,  1998;  Mayer,  2007],  it  is  not 
appropriate  for  situations  involving  great  uncertainty,  as  it  assumes  a  predetermined 
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path  through  an  established  set  of  alternatives.  This  is  antithetical  to  flexibility,  so  a 
different  method  is  needed  [Ekstrom,  2005;  Banerjee,  2004]— one  that  can  take  more 
decision  options  into  account  [Collopy,  2009]. 

Real  Options 

Enter  the  method  of  real  options,  which  exists  at  the  intersection  of  value  and 
uncertainty.  Economic  theory  defines  real  options  as  the  “right,  but  not  the  obligation 
to  take  an  action  at  a  predetermined  cost  and  at  a  predetermined  time”  [Shah,  2008]. 
Traditional  methods  to  cope  with  the  future  uncertainty  are  mainly  focusing  on  making 
accurate  forecast  for  future  uncertainty  and  preparing  for  it.  However,  present  decision 
making  environment  is  not  only  uncertain  but  also  dynamic.  Even  though  the 
forecasting  was  accurate  in  the  past,  it  might  not  be  valuable  because  of  the  change  of 
environment.  One  of  the  efficient  ways  to  react  to  dynamic  uncertain  situation  is  using 
real  option.  The  terminology,  “real  option”,  implies  that  it  is  a  counter  part  of  “financial 
option”  and  the  option  is  not  traded  in  the  financial  market.  As  a  result,  the  real  option 
valuation  often  refers  to  the  well-developed  financial  theory. 

Because  the  system  flexibility  provides  decision  makers  the  ability  to  cope  with  future 
uncertainty,  it  has  value.  The  valuation  can  be  categorized  into  two  groups;  absolute 
valuation  and  relative  valuation.  Absolute  valuation  purely  focuses  on  the  elements  of 
the  real  option  such  as  the  stochastic  behavior  of  underlying  asset.  In  contrast,  relative 
valuation  method  concentrates  on  the  relative  value  of  the  option  rather  than  the  value 
of  itself. 

The  absolute  valuation  approach  model  the  decision  environment  with  stochastic 
dynamic  system,  for  example,  stochastic  differential  equations.  The  value  of  flexible 
decision  opportunity  can  be  obtained  by  solving  the  system.  The  stochastic  dynamic 
programming  was  developed  by  Richard  Bellman  and  others  in  the  1950s.  To  solve  the 
dynamic  equation,  boundary  conditions  are  needed.  Samuelson  [1965]  provided  a  useful 
boundary  condition  as  known  as  ‘smooth  pasting  condition’,  in  the  context  of  economic 
decision  making. 

The  relative  valuation  is  also  called  as  risk-neutral  valuation  or  contingent  claim 
analysis.  The  idea  of  this  approach  is  based  on  the  two  assumptions.  First,  there  are 
other  investment  opportunities  whose  values  are  known.  Second,  a  synthetic  portfolio 
which  is  identical  to  the  real  option  can  be  constructed  using  the  known  opportunities. 
The  theoretical  background  of  relative  valuation  can  be  found  in  (Arrow  1970).  From  the 
seminal  works  of  Black  and  Scholes  [1973],  Merton  [1971]  and  Merton  [1973],  relative 
valuation  has  been  the  paradigm  of  finance.  Cox  and  Ross  [1976]  and  Cox  et  al.  [1979] 
developed  clarified  the  random  walk  representation  of  Brownian  motion  and  used  it  for 
contingent  claims  valuation.  The  well-known  numerical  valuation  method,  ‘binomial 
tree’,  was  suggested  in  this  work.  A  rigorous  statement  of  risk-neutral  valuation  was 
provided  by  Harrison  and  Kreps  [1979].  In  terms  of  mathematics,  contingent  claims 
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analysis  works  under  the  assumption  that  uncertainty  over  the  payoff  from  the 
investment  is  spanned  by  existing  assets.  Duffie  and  Huang  [1985]  investigated  the 
conditions  required  for  dynamic  spanning. 

Because  of  analogous  between  financial  options  and  real  options,  the  evaluation  method 
of  financial  option  is  widely  used  for  evaluation  of  real  options.  The  relationship 
between  financial  options  and  investment  decisions  has  been  an  interesting  issue.  From 
a  practical  point  of  view,  Mun  [2006]  provides  summaries  of  the  differences  between 
financial  options  and  real  options  in  the  following  table. 


Financial  Options 

Real  Options 

Short  maturity 

Long  maturity 

Underlying  variable  driving  its  value 
is  equity  price  or  price  of  a  financial 
asset 

Underlying  variables  are  free  cash 
flows,  which  in  turn  are  driven  by 
competition,  demand,  management. 

Cannot  control  option  value  by 
manipulating  stock  prices 

Can  increase  strategic  option  value 
by  management  decisions  and 
flexibility 

Values  are  usually  small 

Major  million  and  billion  dollar 
decisions 

Competitive  or  market  effects  are 
irrelevant  to  its  value  and  pricing 

Competition  and  market  drive  the 
value  of  a  strategic  option. 

Have  been  around  and  traded  for 
more  than  three  decades 

A  recent  development  in  corporate 
finance  within  the  last  decade. 

Usually  solved  using  closed-form 
partial  differential  equations  and 
simulation/variance  reduction 
techniques  for  exotic  options. 

Usually  solved  using  closed  form 
equations  and  binomial  lattices  with 
simulation  of  the  underlying 
variables,  not  on  the  option  analysis. 

Marketable  and  traded  security  with 
comparables  and  pricing  info 

Not  traded  and  proprietary  in 
nature,  with  no  market  comparables. 

Management  assumptions  and 
actions  have  no  bearing  on  valuation 

Management  assumptions  are 
actions  drive  the  value  of  a  real 
option 

Table  13:  Summary  of  the  Difference  BetweenFinancial  Options  and  Real  Options 
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There  is  interesting  work  related  to  real  options.  The  following  are  well-known 
examples. 

Myers  [1977]  showed  that  firms’  investment  options  are  a  component  of  their  market 
value.  Mcdonald  and  Siegel  [1986]  examined  when  the  optimal  timing  of  investment  is. 
They  considered  the  case  that  the  cost  of  investment  is  onetime  and  fixed,  and  value  of 
the  investment  follows  a  stochastic  process.  Marcus  and  Modest  [1984]  considered  the 
case  with  operating  costs  in  the  context  of  agricultural  production  decisions.  Mcdonald 
and  Siegel  [1985]  shows  that  if  price  follows  a  geometric  Brownian  motion,  a  unit  output 
project  with  fixed  operating  cost  can  be  valued  as  the  sum  of  an  infinite  set  of  European 
call  options. 

When  the  price  or  utility  of  a  project  follows  a  geometric  Brownian  motion,  the 
difference  between  the  expected  rate  of  price  growth  and  the  risk-adjusted  expected 
return  has  important  meaning.  Mathematically  it  is  one  of  the  conditions  for  existence 
of  solution  in  the  infinite  horizon  problem  [Dixit  and  Pindyck  1994].  Economic  meaning 
of  this  difference  was  investigated  by  Mcdonald  and  Siegel  [1984].  When  the  commodity 
is  storable,  the  convenience  yield  which  accumulates  to  inventory  holders  should  reflect 
the  difference  between  two  rated.  Gibson  and  Schwartz  [1990]  and  Brennan  [1991] 
examined  the  stochastic  structure  of  convenience  yield. 

Entry  or  exit  from  an  industry  and  postpone  or  resume  a  task  are  important  issues  in  the 
Real  Options  Theory.  Brennan  and  Schwartz  [1985]  provide  a  general  model  of  the 
decision  to  open,  close  and  postpone  a  mining  project  whose  price  fluctuates  over  time. 
Dixit  [1989]  examined  the  decision  of  entry  and  exit. 

Switching  among  alternatives  is  a  traditional  topic  in  real  options.  In  the  financial 
economics,  Geske  [1979],  Geske  and  Johnson  [1984]  and  Carr  [1988]  researched  the 
optimal  switching  among  a  number  of  choices  according  to  the  change  of  economic 
conditions.  Under  appropriate  assumptions,  the  combination  of  switching  can  be 
interpreted  as  a  set  of  compound  options.  In  the  real  options  context,  Fine  and  Freund 
[1990]  studied  a  general  two-period  model  with  the  choices  of  low  cost  specific  capital  or 
high  cost  all-purpose  capital.  He  and  Pindyck  [1992]  considered  the  subsequent 
expansion  of  capacity.  Bentolila  and  Bertola  [1990]  investigated  the  optimal 
employment  level  with  hiring  and  firing  costs.  Kogut  and  Kulatilaka  [1994]  examined 
the  choice  of  multinational  company  that  switching  production  from  one  country  to 
another  country  according  to  the  fluctuation  of  exchange  rate. 

Majd  and  Pindyck  [1987]  provided  a  continuous  investment  and  time  to  build  model. 
They  modeled  the  case  that  a  firm  invests  continuously  until  the  project  is  completed, 
and  investment  can  be  stopped  and  restarted  later  without  any  cost. 

Gaps  between  practice  and  current  research 


Contract  Number:  H98230-08-D-01 71 

Report  No.  SERC-2010-TR-010 
09/25/2010 


DO  02,  TO  02  RT  018 


UNCLASSIFIED 

107 


UNCLASSIFIED 


The  dynamic  programming  approach  usually  assumes  risk-neutral  decision  makers  and 
uses  the  risk-free  rate  for  the  discount  rate.  However,  if  the  decision  maker  is  risk 
averse,  it  is  necessary  to  either  use  an  appropriate  discount  rate  or  to  use  a  certainty 
equivalent  benefit. 

The  relative  pricing  approach  may  not  work  in  some  situations.  For  example,  when 
qualitative  characters  of  the  choices  are  important,  risk-neutral  valuation  does  not  work. 
Suppose  that  the  CEO  of  Ferrari  is  considering  expanding  the  factory  to  manufacture 
the  Ferrari  with  new  technology.  To  evaluate  the  value  of  this  investment  opportunity, 
he  constructs  a  synthetic  portfolio  of  a  brand-new  factory  in  a  developing  country,  such 
as  China,  and  bonds.  Maybe  the  monetary  values  of  both  investments  are  identical. 
However,  the  Ferrari  which  is  manufactured  in  the  Chinese  factory  may  not  be  the 
Ferrari  that  we  know  so  far.  To  evaluate  investment  opportunities,  we  can  use  a 
common  unit,  such  as  U.S.  dollars.  However,  a  common  measure  does  not  guarantee 
that  the  opportunities  are  exchangeable.  Furthermore,  is  the  money  that  comes  from  a 
developing  country  different  then  the  money  that  comes  from  a  developed  country?  Can 
we  distinguish  the  cash  generated  by  investing  to  a  stock  or  a  bond?  For  the  finance 
field,  relative  pricing  works  well.  But  when  we  consider  real-option,  we  need  to  check 
the  qualitative  property  of  the  opportunity. 

In  addition,  note  that  relative  pricing  works  only  for  complete  markets.  In  a  complete 
market,  every  asset  can  be  perfectly  replicated  with  other  assets  in  the  market.  It  means 
that  the  totally  “NEW”  asset  which  is  not  replicable  with  existing  assets  cannot  be 
evaluated  with  risk-neutral  valuation. 

There  are  a  lot  of  scholars  stating  that  the  contingent  claim  can  be  priced  through  any 
asset  regardless  whether  its  synthetic  portfolio  is  traded  in  the  market  or  not.  Among 
those,  Harrison  and  Kreps  [1979]  provide  a  rigorous  mathematical  proof,  and  Cox, 
Ingersoll  et  al.  [1985]  also  concur  with  this  opinion.  However  all  of  these  works  assume 
the  complete  market  and  do  not  consider  the  qualitative  aspect  of  choice. 

The  binomial  tree  method  is  the  most  popular  numerical  method  to  evaluate  the  value  of 
options.  Most  of  binomial  tree  methods,  for  example,  Cox,  Ross  et  al.  [1979]  implicitly 
assume  risk-neutral  valuation.  We  need  to  be  careful  before  using  the  binomial  tree 
method  for  evaluating  real  options. 

Copeland  [1998]  claims  that  only  real  options  can  “provide  a  theoretically  sound  tool  for 
valuing”  decision  flexibility.  In  a  manufacturing  application,  Ajah  [2010]  touts  real 
options,  “which  incorporates  uncertainty  in  a  theoretically  consistent  manner,”  and 
“that  the  adoption  of  the  real  options  approach  early  in  the  conceptual  design  process 
can  offer  to  the  designer,  extra  degrees  of  freedom  of  systematically  considering  and 
designing  system  elements.”  In  an  article  about  flexible  on-orbit  satellite  servicing, 
Joppin  [2003]  uses  the  case  of  a  satellite  upgrade  to  create  a  dynamic  framework  based 
on  real  options  theory  to  capture  the  flexibility  of  the  on-orbit  servicing  paradigm.  In 
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the  information  technology  domain,  authors  have  also  used  the  real  options  technique 
in  an  effort  to  quantify  flexibility  [Ekstrom,  2005;  Kumar,  1999]. 

Another  example  where  real  options  analysis  is  used  to  help  account  for  the  value  of 
flexibility  comes  from  the  Defense  Advanced  Research  Project  Agency  (DARPA).  With 
the  spacecraft  development  program  known  as  F6  providing  the  backdrop,  Brown 
[2008]  discusses  the  need  to  calculate  the  variance  in  net  value  of  a  given  architecture  in 
order  to  perform  architectural  trades.  This  goal  is  essential  to  the  F6  program,  which 
proposes  a  revolutionary— and  truly  flexible— satellite  architecture  consisting  of  clusters 
of  satellite  modules,  physically  “fractionated,”  but  connected  wirelessly  so  as  to  be 
functionally  similar  to  traditional  monolithic  satellites.  To  be  able  to  compare  the  value 
of  the  two  disparate  architectural  options,  Brown  uses  real  options  theory,  and  employs 
the  time-honored  Black-Sholes  model  to  mathematically  express  the  value  of  various 
design  options. 

Collopy  [2009]  also  advocates  the  use  of  real  options  as  a  decision-making  model,  but 
avers  that  Black-Scholes  is  only  legitimate  under  certain  conditions:  “(a)  the  underlying 
asset  follows  a  geometric  random  walk,  which  is  to  say  its  motions  fall  into  a  lognormal 
distribution,  and  movements  in  non-overlapping  periods  are  uncorrelated;  and  (b)  the 
underlying  asset  is  traded  on  an  efficient  market,  that  is,  a  market  in  which  there  is  no 
possibility  for  arbitrage.”  Collopy  then  notes  that  the  second  assumption  contradicts 
real  options,  in  general,  and  the  first  assumption  does  not  apply  to  the  defense  industry. 
So,  if  we  elect  to  use  real  options  analysis  in  valuing  flexibility,  Black-Scholes  may  not  be 
the  appropriate  model. 

The  F6  example  highlights  the  two  principal  drawbacks  with  real  options  theory.  Like 
any  valuation  technique,  we  must  establish  the  criteria  for  evaluation,  which  is  not 
always  straightforward.  For  the  F6  program,  the  attributes  selected  as  evaluation 
criteria  included  the  degree  of  fractionation,  the  reliability  of  each  module,  and  the 
modes  of  connectivity  between  the  modules.  The  specific  value  assignments  are  then 
largely  determined  through  stakeholder  interviews.  This  infuses  a  high  degree  of 
subjectivity  into  the  process,  which  relates  to  the  second  challenge  of  real  options.  As  a 
predictive  model,  it  is  only  as  good  as  its  inputs.  If  the  inputs  are  of  questionable 
validity,  then  so  too  will  be  the  results.  Also  note  that  real  options  doesn’t  get  us  much 
closer  to  our  final  objective.  While  it  can  be  useful  if  one  knows  the  current  cost  of  the 
option  and  the  expected  value  at  a  later  time,  the  fact  is  that,  for  design  flexibility,  we 
lack  knowledge  for  both  elements  (although  one  could  argue  that  we  should  know  the 
current  cost). 

In  the  upcoming  sections,  we  present  methods  of  quantifying  flexibility  that  use  real 
options  as  the  basis  of  determining  value.  Saleh  [2009]  cautions  against  this.  He  notes 
that  while  real  options  is  useful,  it  simply  cannot  provide  insight  into  how  to  embed 
flexibility  into  the  system,  and  that  many  are  using  it  inappropriately. 
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“It  is  important  to  note  the  difference  between  the  value  of  an  attribute,  here 
flexibility,  and  the  measure  of  that  attribute.  For  example,  how  reliable  a  system  is 
differs  from  how  much  its  reliability  is  worth  ...  Several  authors,  in  their  attempt  to 
bring  the  Real  Options  mindset  to  engineering  design,  fail  to  recognise  this 

distinction  between  the  value  of  an  attribute,  such  as  flexibility,  and  the 
measure  of  that  attribute”  (emphasis  added). 

Saleh’s  observation  is  trenchant.  Indeed,  as  noted  in  the  introduction  to  this  chapter,  we 
will  find  as  we  progress  that  most  authors  do  fail  to  make  this  distinction.  However,  this 
is  not  a  fault  with  real  options,  per  se,  but  rather  an  error  in  its  application.  We  do  need 
to  quantify  the  value  of  flexibility,  and  it  would  appear  that  real  options  analysis  is  a 
vital  piece  of  the  flexibility  puzzle.  We  have  already  noted  the  strong  relationship 
between  flexibility  and  uncertainty,  and  the  principle  of  uncertainty  is  at  the  heart  of 
real  options  analysis.  Illustrating  this  linkage,  one  author  chose  to  essentially  define 
flexibility  as  a  real  option,  i.e.,  “Flexibility  therefore  can  be  seen  as  an  insurance 
premium  which  is  paid  at  present  in  order  to  have  a  possible  advantage  in  [the]  future” 
[Nachtwey,  2009].  Real  options  analysis  techniques  are  therefore  likely  to  be  crucial  in 
the  effort  to  justify  the  investment  in  flexibility. 

The  Option  Space  (Pareto,  DSM,  HOP,  DQDAF) 

One  aspect  of  quantifying  the  value  of  flexibility  is  identifying  and  enumerating  the 
many  choices  that  are  available  to  us  during  the  design  phase.  This  range  of  choices  is 
central  to  the  notion  of  flexibility  [Chen,  1999],  and  it  is  known  as  the  option  space. 
Several  authors  have  discussed  methods  to  characterize  it.  The  most  prominent 
method,  by  far,  is  to  construct  the  Pareto  set  (also  referred  to  as  the  “Pareto  frontier”  or 
the  “Pareto  front”),  which  is  the  full  set  of  optimum  design  points  that  results  when  one 
must  account  for  multiple  competing  objectives  [Olewnik,  2006],  and  thus  represents 
“the  best  achievable  tradeoff  between  capacity  and  lifecycle  cost”  [de  Week,  2003; 
Hollingsworth,  2004]  recommends  the  intuitive  phrase,  a  “trade-off  surface.”  Examples 
of  flexibility  quantification  research  in  which  Pareto  frontiers  are  constructed  include 
[Brown,  2009;  Nilchiani,  2003;  Roser,  1999;  Ross,  2008;  Haubelt,  2002;  Lasserre, 
1985;  Chattopadhyay,  2009;  Brathwaite,  2009]. 

A  challenge  with  this  approach  is  how  to  generate  the  spread  of  points  along  the  Pareto 
frontier,  especially  for  a  large  option  space.  “Because  weighted  sum  methods  have 
difficulty  in  finding  and  generating  Pareto  frontiers,”  other  approaches  have  been 
suggested,  including  genetically-based  evolving  algorithms  [Eddy,  2001].  Eddy  argues 
for  the  suitability  of  this  approach  as  “nearly  all  design  methodologies  incorporate  the 
concept  of  evolving  designs.” 

The  question  of  how  the  Pareto  frontier  relates  to  actual  design  flexibility  becomes  the 
next  concern.  Olewnik,  [2006]  tackles  this  aspect  of  the  problem,  as  part  of  his  research 
effort  to  develop  a  “decision  support  framework  for  the  design  of  flexible  systems.”  One 
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of  his  key  goals  is  to  quantify  flexibility,  and  constructing  the  Pareto  frontier  is  the  first 
step,  which  provides  the  boundary  of  the  design  space:  “The  ideal  flexible  system 
provides  optimal  performance  by  configuring  itself  to  provide  the  performance 
associated  with  the  extreme  points  of  the  Pareto  frontier.”  Olewnik  then  proposes  that 
the  measure  of  flexibility  is  equivalent  to  the  “distance”  between  the  extreme  points  of 
the  Pareto  frontier,  though  it’s  not  clear  why  this  should  be.  Intuitively,  the  distance 
between  extreme  points  on  the  Pareto  frontier  would  seem  to  better  indicate  the  range 
of  flexible  design  options,  vice  the  system’s  actual  flexibility. 

Another  challenge  regarding  the  option  space  is  to  assess  the  nature  and  degree  of 
interdependence  between  multiple  design  considerations  and  requirements.  This  is 
frequently  accomplished  by  use  of  the  Design  Structure  Matrix  (DSM)  methodology. 
The  DSM  was  introduced  three  decades  ago  by  Steward  as  a  ’’framework  for  formally 
identifying  and  tracking  relationships  between  design  variables”  [Steward,  1981].  Keese 
[2007]  and  Qureshi  [2006]  use  DSMs  for  interdependence  analysis  as  part  of  their 
patent  study  work  on  flexibility. 

Since  “a  primary  goal  in  basic  DSM  analysis  is  to  minimize  the  number  of  feedbacks  and 
their  scope  by  restructuring  or  re-architecting  the  process”  [Abdelsalam,  2007],  it’s  no 
surprise  that  this  technique  has  been  used  extensively  in  terms  of  assessing  modularity 
[Lai,  2008;  Sosa,  2005;  Holtta-Otto,  2007;  Clarkson,  2001;  Ethiraj,  2004;  Stryker, 
2009].  Shifting  the  focus  from  workflows  to  information  flows,  a  clearer  understanding 
of  interdependencies  can  emerge.  The  overlap  with  flexibility  is  clear  as  well,  since 
DSMs  can  help  identity  how  change  propagates  through  a  system  [Eckert,  2004]. 
Clarkson  [2001]  utilizes  DSMs  in  this  way  as  a  method  for  change  prediction  within  the 
system  itself,  which  can  help  determine  the  probabilities  across  the  option  space.  In 
Rajan’s  study  examining  the  relationship  between  flexibility  and  the  degree  of 
modularization,  the  DSM  “facilitates  a  complete  view  of  the  product  configuration  in  a 
reasonably  concise  format”  [Rajan,  2005]. 

An  alternative  approach  that  may  be  used  to  identity  design  interdependencies  is  House 
of  Quality  (HOQ).  HOQ  is  the  fundamental  design  tool  of  the  broader  customer  needs 
tool,  Quality  Function  Deployment.  The  fundamental  premise  of  the  HOQ  process  is 
that  all  members  of  the  product  team  need  to  work  together  from  the  outset  in  order  to 
produce  a  product  that  best  meets  the  customer’s  needs  [Hauser,  1988].  By 
constructing  a  matrix  that  resembles  a  house  (to  include  a  roof),  HOQ  “provides  a  fast 
way  to  translate  customer  requirements  into  specifications  and  systematically  flowdown 
the  requirements  to  lower  levels  of  design,  parts,  manufacturing,  and  production” 
[Haskins,  2007].  The  particularly  relevant  component  of  this  process  is  the  Technical 
Correlation  Matrix,  which  documents  the  relationships  between  customer  requirements 
and  system  design  solutions  in  an  effort  to  identify  where  dependencies  and  conflicts  are 
likely  to  exist  or  arise. 
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Another  option  that  may  be  worth  mentioning  is  the  Department  of  Defense 
Architectural  Framework  (DODAF).  The  DODAF  is  a  conceptual  model  that  guides  the 
development  of  system  and  mission  architectures.  It  is  intended  for  use  by  DoD 
managers  at  all  levels  to  make  key  decisions  more  effectively.  Central  to  the  framework 
is  the  concept  of  standardized  products  known  as  “views”  that  allow  for  visualizing, 
understanding,  and  assimilating  the  broad  scope  and  complexities  of  an  architecture 
[Department  of  Defense  2009].  While  there  appears  to  be  no  discussion  in  the 
flexibility  literature  regarding  DODAF,  it  is  feasible  that  certain  architectural  views  may 
be  applicable  in  our  attempt  to  characterize  design  interdependencies. 

There  are  two  particular  DODAF  views  that  may  be  of  use.  The  first  is  System  View-5a 
(SV-5a),  Operational  Activity  to  System  Function  Traceability  Matrix.  The  SV-5a  maps 
system  functions  back  to  operational  activities,  thereby  identifying  transformation  of  an 
operational  need  into  a  purposeful  action.  During  requirements  definition,  the  SV-5a 
plays  a  key  role  in  tracing  architectural  elements  associated  with  system  function 
requirements  to  those  associated  with  user  requirements.  The  second  potential  view  of 
interest  is  the  complementary  SV-5b,  Operational  Activity  to  System  Traceability 
Matrix.  This  view  maps  various  systems  back  to  capabilities  or  operational  activities, 
which  serves  to  transform  operational  needs  into  purposeful  actions  performed  by  the 
system. 

The  SV-5  is  certainly  not  a  mathematically-based,  rigorous  method  of  transformation. 
In  addition,  it  is  more  applicable  to  higher  levels  of  abstraction  such  as  the  design  of 
systems-of-systems.  Nevertheless,  the  SV-5  construct  and  terminology  are  familiar  to 
DoD  program  managers,  and  thus,  may  be  well  suited  for  depicting  the  results  of  a  more 
formal  mathematical  transformation. 

Characterizing  the  option  space  is  a  vital  element  of  any  attempt  to  quantify  the  value  of 
flexibility.  Tools  and  methods  such  as  Pareto  fronts,  DSMs,  and  HOQ  are  featured 
prominently  in  the  literature  as  ways  to  identify  the  boundaries  and  interdependencies 
of  the  design  option  space. 

Value-Driven  Design  (and  Decision-Based  Design) 

It  perhaps  goes  without  saying  that  the  notion  of  value  with  respect  to  the  system  design 
is  central  to  the  goal  of  quantifying  the  value  of  flexibility.  Therefore,  of  prime  interest 
to  us  is  the  principle  of  value-driven  design  (VDD),  which  attempts  to  incorporate  value 
metrics  into  systems  engineering  design.  VDD  is  a  movement  to  refocus  systems 
engineering  processes  on  the  optimization  of  the  overall  system  design,  vice  the 
optimization  of  specific  system  performance  parameters.  There  is  growing  interest  and 
research  into  VDD  across  industry  and  the  DoD  [Collopy,  2008].  Owen  Brown  employs 
VDD  on  F6,  and  provides  an  excellent  summary  of  the  concept  (which  he  refers  to  as 
value-centric  design ): 
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“In  traditional  requirements-driven  systems  engineering  process,  design  choices  are 
based  on  whether  or  not  the  outcome  will  meet  the  requirements.  All  designs  that  meet 
requirements  are  equally  good.  All  designs  that  fail  to  meet  requirements  are  equally 

bad .  Value-Centric  Design  chooses  the  best  design  whether  or  not  individual 

attributes  exceed  a  threshold”  [Brown,  2009]. 

This  distinction  is  likely  to  be  extremely  important  if  we  are  to  allocate  value  to  a  non- 
traditional  system  characteristic  like  flexibility.  To  implement  VDD,  we  must  first 
develop  a  value  model,  which  is  the  objective  function  for  comparing  the  worth  of  one 
design  to  another  [Collopy,  2009].  Value  models  are  suitable  for  capturing  the  upside  of 
uncertainty,  departing  from  the  standard  reckoning  of  only  the  downside  uncertainty  via 
traditional  risk  management  techniques.  Thus,  a  VDD  model  could  feasibly  capture  the 
value  added  by  investing  in  a  more  flexible  design  because  of  the  potential  payoff  later. 
Furthermore,  if  the  value  of  flexibility  can  be  quantified  in  units  that  are 
commensurable  with  cost,  then  meaningful  cost -value  tradeoffs  can  be  made.  In  other 
words,  VDD  may  help  us  with  the  critical  task  of  directly  comparing  costs  and  benefits 
by  assigning  values  to  each  parameter  that  have  the  same  units  of  measurement 
(presumably  dollars).  Then  the  best  (i.e.,  most  cost-effective)  design  is  simply  the  one 
with  the  highest  expected  utility  [Brown,  2009]. 

Collopy  provides  the  following  explanation  to  help  the  reader  envision  the  value  model 
and  how  it  is  used: 

“One  way  to  think  of  the  model  is  in  terms  of  a  high  dimensional  attribute  space,  where 
every  attribute  is  an  orthogonal  coordinate  axis  of  the  space.  Each  design  maps  to  a 
point  in  the  space.  The  value  model  is  a  potential  function  on  the  space.  The  model 
assigns  a  single  scalar  value  to  every  point  in  the  attribute  space,  much  as  an  electric 
field  assigns  to  each  point  in  physical  space  a  voltage  (electrical  potential),  or  a  flow 
field  assigns  to  each  point  a  pressure.  The  mapping  forms  a  value  surface  over  the 
space.  Design  changes  that  move  the  design  up  the  value  surface  are  good,  even  if  they 
detract  from  some  properties”  [Collopy,  2003]. 

Again,  we  see  this  idea  of  increasing  the  aggregate  value  of  the  system,  even  at  the  cost 
of  local  performance  decreases.  Value-driven  design  is  not  yet  used  routinely  in  the 
defense  industry,  but  the  underlying  principle  represents  a  potential  sea-state  change  in 
how  the  DoD  would  manage  its  programs.  The  fact  is  that  the  cost  to  develop  and 
operate  a  system  that  meets  certain  requirements  over  a  given  time  period  is  not  a 
deterministic  value.  By  valuing  non-traditional  system  characteristics  like  flexibility, 
VDD  essentially  provides  a  more  efficient  and  strategic  approach  to  systems  acquisition, 
providing  a  more  accurate  and  integrated  assessment  of  true  system  cost  than  other 
extant  putative  measures  of  life-cycle  cost.  Brown  calls  this  new  proposed  measure  of 
true  overall  system  cost  the  “stochastic  lifecycle  cost”  [Brown,  2007]. 
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Virtually  all  articles  that  claim  to  quantify  flexibility  in  system  design  are  employing 
VDD  principles  to  some  degree  (e.g.,  [Nilchiani,  2006;  Ajah,  2010;  Banerjee,  2004; 
Joppin,  2003;  Hazelrigg,  1998;  Ekstrom,  2005]).  Some  approaches  are  straightforward, 
and  some  are  complex,  even  convoluted.  Ultimately,  though,  the  task  is  simple:  We  are 
presented  with  a  cost-optimization  problem  whereby  we  must  obtain  the  highest  design 
value  for  the  lowest  investment  cost.  The  problem,  of  course,  is  that  neither  side  of  the 
equation  has  been  satisfactorily  resolved.  Scholars  rarely  tackle  the  cost  side  of  the 
equation,  i.e.,  the  question  of  how  much  does  it  cost  to  implement  design  flexibility. 
And  there  are  equally  few,  if  any,  approaches  to  address  the  value  side  of  the  equation, 
at  least  in  monetizable  metrics.  Put  bluntly,  we  lack  a  consistent,  practical,  and 
valid  method  for  assigning  the  value  for  each  flexibility  option. 

In  fairness,  there  was  one  attempt  to  do  exactly  that.  Acknowledging  that  an 
appropriate  value  measure  of  flexibility  must  consider  uncertainty  and  risk,  Olewnik 
[2006]  employs  the  Decision-Based  Design  principle  [Hazelrigg,  1998]  and  utility 
theory  to  assign  value  to  each  of  the  design  options.  Value  assignments  are  achieved 
through  a  method  of  Consumer  Choice  Theory  known  as  Conjoint  Analysis.  Selecting 
the  best  option  then  simply  entails  finding  the  alternative  that  has  the  highest  expected 
utility.  Olewnik’s  approach  is  intriguing,  but  his  proposed  method  of  valuation  is  far 
more  applicable  to  a  classical  profit-centered  approach,  which  is  not  the  motivating 
principle  for  defense  systems.  It  should  also  be  noted  that  Olewnik’s  conception  of 
flexibility  is  much  more  in  line  with  our  definition  of  adaptability,  even  describing  the 
“ideal  flexible  system  is  one  where  all  of  the  variables  are  (potentially)  actively 
adaptable”  [Olewnik,  2006].  Nevertheless,  it  may  be  useful  as  a  starting  point  in  our 
efforts. 

Filtered  Outdegree 

Now  that  we’ve  completed  a  basic  review  the  applicable  tools,  processes,  and  methods, 
we  can  commence  an  in-depth  examination  of  specific  methods  that  purport  to  quantify 
flexibility. 

Under  Ross’  changeability  framework,  flexibility  is  indirectly  derived  through  the 
quantification  of  changeability.  In  order  to  quantify  changeability,  Ross  first  creates  a 
tradespace  framework  consisting  of  so-called  “change  events”  that  are  characterized  by 
three  change  elements: 

•  Change  Agent:  The  force  that  causes  the  change.  Recall  from  earlier  that  Ross 
discriminates  between  adaptability  and  flexibility  based  on  the  location  of  the 
change  agent  (i.e.,  internal  change  agents  pertain  to  adaptability;  external  change 
agents  pertain  to  flexibility). 

•  Change  Effect:  The  difference  in  states  before  and  after  a  change  has  taken  place. 
Recall  that  Ross  identifies  three  categories  of  effects:  robustness,  scalability,  and 
modifiability. 
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•  Change  Mechanism:  The  particular  path  the  system  must  take  in  order  to 
transition  from  its  original  state  to  its  changed  state.  The  more  change  paths  that 
a  system  may  traverse,  the  more  “changeable”  it  is. 

According  to  this  framework,  each  change  effect  essentially  represents  a  different 
system  design  approach.  In  order  to  assess  which  design  approach  is  preferred,  Ross 
calls  for  a  measure  of  “goodness”  to  rank  alternatives,  preferably  a  relatively  rigorous 
measure,  such  as  a  multiattribute  utility  function.  Like  many  other  researchers,  Ross 
generates  a  Pareto  frontier  of  the  highest  value  designs. 

Graphically  depicted,  nodes  can  be  used  to  designate  system  design  options,  with  arcs 
drawn  to  denote  transition  paths.  Each  arc  would  correspond  to  a  particular  system 
design  (i.e.,  potential  change  mechanism),  and  have  an  associated  cost  (in  terms  of 
dollars  and  time).  Ross  then  introduces  the  term,  “out degree,”  which  he  defines  simply 
as  the  number  of  outgoing  arcs  from  a  given  design.  Each  design  has  a  particular 
outdegree  number,  which  represents  an  objective  value  of  that  design,  and  “provides  a 
mechanism  for  system  designers  to  explicitly  improve  the  potential  changeability  of  a 
system.  Ross  refines  the  outdegree  concept  further  by  establishing  the  “filtered 
outdegree,”  which  is  the  number  of  outgoing  arcs  for  a  particular  design  where  the  cost 
is  less  than  the  acceptability  threshold  of  a  given  decision-maker.  Using  this  approach, 
we  can  then  measure  flexibility  of  a  given  design  by  calculating  the  filtered  outdegree, 
but  only  counting  the  change  mechanisms  caused  by  external  change  agents. 

The  proposed  framework  is  compelling,  as  it  provides  a  core  mechanism  for  assessing 
how  much  investment  is  necessary  to  obtain  a  certain  amount  of  flexibility,  and  is  to  be 
commended  for  considering  the  cost  side  of  the  value  equation.  But  it  is  also  missing 
some  key  elements,  including  how  to  actually  value  changeability  and  flexibility  (e.g., 
NPV,  real  options),  as  well  as  how  to  establish  the  cost  of  each  transition  path.  Ross  was 
aware  of  at  least  the  former  omission,  describing  it  as  a  deliberate  decision  due  to  the 
vagaries  of  valuation.  In  his  words,  “Valuation  of  ilities  as  a  single  metric  is  an 
additional  layer  of  analysis  that  can  be  put  on  top  of  the  proposed  framework,”  but  since 
“all  valuation  techniques  rely  upon  specific  assumptions  regarding  how  to  collapse  time, 
utility,  cost,  and  uncertainty  into  a  single  metric,”  inclusion  would  “reduce  the 
generalizeability  of  the  framework.”  This  article  also  does  not  provide  any 
recommendations  on  how  to  identify  potential  transition  paths,  which  is  a  nontrivial 
aspect  of  the  design  challenge. 

Shah  [2008]  makes  use  of  the  outdegree  approach  as  well,  while  also  filling  in  some  of 
the  research  holes.  Shah  uses  a  DSM-like  tool  (which  he  calls  the  Engineering  System 
Matrix)  to  “find  useful  points  to  insert  options  in  the  physical  architecture,”  thereby 
providing  some  guidance  on  identifying  potential  transition  paths.  Shah  also  endorses 
the  use  of  real  options  to  help  determine  the  relative  value  of  the  various  design  options. 
Shah’s  view  is  that  if  the  outdegree  of  a  given  design  is  perceived  to  be  too  inflexible,  or 
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too  costly,  then  embedding  the  opportunity  for  real  options  into  the  design  may  improve 
the  cost/benefit  ratio,  thus  justifying  the  investment. 

Shah  also  endorses  a  second  technique  for  identifying  aspects  of  the  design  that  are 
most  likely  to  change  or  be  impacted  by  change.  This  technique,  developed  by  Eckert 
[2004],  investigates  the  role  of  modularity  and  its  ability  to  stem  the  propagation  of 
change.  Eckert  devises  a  metric  called  the  Change  Propagation  Analysis  (CPA)  as  a 
means  of  predicting  the  effects  on  the  system  of  planned  and  unplanned  changes.  CPA 
can  help  identify  system  components  that  tend  to  absorb  or  magnify  changes  and  thus 
allow  “the  designer  to  investigate  how  possible  changes  will  impact  the  structure  and 
behavior  of  a  system  design.”  By  combining  DSM  techniques  to  identify  design  and 
change  interdependencies,  filtered  outdegree  to  determine  changeability,  and  real 
options  to  provide  the  contextual  value,  Shah  seems  to  provide  one  of  the  most 
comprehensive  approaches  to  quantifying  (and  valuing)  flexibility. 

Most  recently,  the  filtered  outdegree  method  has  been  refined  still  further  by  a  colleague 
of  Ross.  Viscito  [2009]  coins  the  phrase,  Value  Weighted  Filtered  Outdegree  (VWFO), 
which  is  “a  metric  that  captures  the  utility  difference  between  an  originating  design  and 
its  possible  destination  designs.”  Importantly,  the  VWFO  must  be  defined  in  terms  of 
the  consecutive  time  periods  of  “fixed  context  and  expectations”— called  an  “epoch”— 
that  are  then  encapsulated  into  a  time-ordered  series  known  as  an  “era.”  Each  era 
represents  one  possible  timeline  for  the  system,  and  can  supposedly  facilitate  discrete 
analysis  of  system  flexibility  via  a  computer-based  modeling  technique  that  is  said  to 
enable  evaluation  of  very  large  numbers  of  discrete  designs  in  utility-cost  space.  Within 
this  step,  yet  another  technique  is  introduced  known  as  Tradespace  Network  Analysis 
that  is  intended  to  assess  a  system’s  ability  to  change  states.  When  all  these  factors  are 
integrated,  the  VWFO  is  used  to  identify  those  designs  with  the  highest  utility,  and  thus 
the  most  “valuably  flexible.” 

In  all,  the  filtered  outdegree  research  seems  highly  relevant  to  our  established  goal  of 
quantifying  the  value  of  flexibility.  The  only  weakness  seems  to  be  that  it  will  not  work  if 
the  decision-maker  cannot  specify  where  he  wants  flexibility.  Consider  the  following 
pronouncement  by  Shah:  “If  a  decision  maker  desires  the  system  to  be  flexibly  scaleable 
[sic]  in  image  resolution,  then  only  change  mechanisms  that  result  in  a  change  in  level 
of  image  resolution  performance  will  be  counted  towards  the  calculation  of  the 
outdegree  of  designs  in  the  tradespace  network.”  But  what  if  we  simply  don’t  know 
which  change  mechanisms  will  do  that?  In  reality,  we  often  lack  the  ability  to  cleanly 
map  the  design  space  attributes  onto  the  performance  space.  Ross  seems  aware  of  this 
potential  criticism,  and  evasively  stipulates  that,  “Desiring  “flexibility”  in  a  general  sense 
is  meaningless  from  a  design  perspective,  since  it  is  an  inherently  ambiguous  term,  and 
must  be  further  specified  in  order  to  be  quantifiable  and  testable  as  a  design  goal.” 

Change  Potential  Number 
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Driven  by  the  need  to  characterize  unpredictable  change  in  a  meaningful  way  that 
facilitates  analysis  Rajan  [2003],  proposes  adapting  the  established  method  of 
calculating  Failure  Mode  and  Effects  Analysis  (FMEA)  for  this  purpose.  The  idea  is  to 
take  FMEA’s  systematic  approach  to  identifying  potential  failure  modes,  and  apply  it 
instead  to  look  for  possible  changes  that  may  occur  in  the  system  under  investigation. 
Rajan  dubs  this  modified  process  Change  Mode  and  Effects  Analysis,  or  CMEA. 

The  first  step  in  the  CMEA  is  to  decompose  the  system  “in  some  rational  manner  so  that 
it  can  be  assessed  for  possible  changes.”  Then,  one  must  create  a  CMEA  table  in  order 
to  obtain  the  “Change  Potential  Number”  (CPN)  for  the  system  (compare  to  RPN  in  an 
FMEA).  The  CPN  is  supposed  to  provide  an  indication  of  how  easily  a  change  can  be 
incorporated,  and  is  described  as  “the  overall  flexibility  for  a  given  change.”  Continuing 
to  borrow  from  the  established  FMEA  structure  and  lexicon,  Rajan  uses  three  factors  to 
calculate  a  system’s  CPN. 

•  F :  The  inherent  flexibility  of  a  design  for  a  given  change 

•  O:  The  probability  that  the  change  will  occur 

•  R:  The  Readiness  of  the  developer  to  react  to  the  change 

•  N:  Max  of  (number  of  potential  Change  modes,  number  of  potential  effects  of 
change,  number  of  potential  causes  of  change) 

Each  factor,  F,  R,  and  N,  is  subjectively  evaluated  on  an  interval  scale  from  one  to  ten. 
CPN  is  then  calculated  using  the  following  formula: 

N 

1  V"  \-(Ri  +  Fi)  -  °i  +  8] 

CPN  =  -  >  — - - - - - - 

N  Z-i  27 

i=i 

To  his  credit,  Rajan  then  selects  ten  consumer  products  as  case  studies  to  validate  his 
model  (a  step  omitted  all  too  often  in  this  literature).  The  ten  products  consisted  of 
flexible  products  (i.e.,  “multiple  external  modules)  and  inflexible  products  (i.e.,  “more 
integral  design  with  fewer  modules’).  After  each  product  was  decomposed  in  terms  of 
its  modules/parts,  subjective  scores  were  assigned— though  only  for  one  CPN  factor  (F, 
its  inherent  flexibility— more  on  why  in  a  moment).  Then  the  CPN  score  was  calculated. 
Rajan  considers  the  model  validated  at  this  step  because  the  CPN  scores  for  the 
products  indicated  relative  levels  of  flexibility  consistent  with  the  judgment  of 
“experienced  designers.” 

Other  authors  have  used  the  CMEA  process  when  attempting  to  validate  their  research, 
including  Qureshi  [2006]  and  Keese  [2007].  Finally,  it  may  be  worth  noting  that  the 
research  on  CMEA  has  apparently  not  extended  beyond  its  originating  university 
(University  of  Texas  at  Austin). 
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The  CPN  metric,  as  described  by  Rajan,  raises  a  number  of  concerns,  both  with  the 
methodology,  as  well  as  the  underlying  theory.  In  terms  of  methodology,  the  probability 
that  the  change  will  occur  (O),  and  the  readiness  of  the  developer  to  react  to  the  change 
(R),  both  seem  like  they  would  be  difficult  to  quantify,  and  the  author  doesn’t  explain 
how  it  might  be  done,  even  sidestepping  the  problem  entirely  by  only  considering  the 
inherent  flexibility  factor  (F)  in  his  case  study.  Moreover,  the  validation  approach 
manages  to  be  both  useless  and  invalid  at  the  same  time.  Rajan  conducts  a  functional 
decomposition  based  on  modules,  and  then  selects  a  series  of  products  whose  flexibility 
is  differentiated  based,  in  essence,  on  the  number  of  modules.  Then  he  chooses  to  omit 
the  other  CPN  measures  that  he  devised  because  obtaining  them  would  require  “a 
significant  level  of  industrial  interaction.”  It  is  no  wonder  that  the  results  “validated” 
the  model,  as  the  model  did  little  more  than  scale  the  input  and  call  it  an  output. 
Furthermore— and  perhaps  most  concerning  from  a  theoretical  perspective— Rajan 
provides  no  rationale  for  the  CPN  formula.  It  is  presented,  fully  formed,  without  any 
hint  of  derivation  or  justification,  so  it’s  impossible  to  discern  what  theoretical 
underpinning,  if  any,  applies. 

Given  these  problems,  we  are  left  to  wonder  whether  CPN  is  merely  a  difficult  and 
subjective  measure  of  product  flexibility,  or  wholly  illegitimate. 

Flexibility  Aspects 

Fitzgerald  [2009]  studies  flexibility  in  the  manufacturing  and  information  technology 
fields,  and  proposes  a  method  to  examine  the  “flexibility  aspects”  of  systems  in  terms  of 
the  system’s  potential  flexibility  capabilities  and  the  systems  behavior  under  change.  To 
obtain  a  system’s  flexibility  aspects,  the  system  must  be  decomposed  along  three  distinct 
“flexibility  dimensions.” 

•  Range:  Measures  the  variety  of  alternatives  for  a  given  change,  i.e.,  the  “action 
space” 

•  Response:  The  preparation  time/cost  for  coping  with  a  change  in  the  action 
space 

•  Distension:  The  invested  effort/time/cost  for  enhancing  the  current  action  space 
if  needed  thus  enabling  the  generic  object  to  accommodate  a  given  change  whose 
handling  is  currently  outside  the  action  space. 

Note  that  under  this  conception,  Fitzgerald’s  use  of  “response”  is  akin  to  “versatility”  in 
that  it  measures  the  effort  necessary  to  employ  a  latent  capability,  not  develop  a  new 
one.  Whether  this  is  truly  a  system  change  is  debatable,  and  it  certainly  isn’t  consistent 
with  most  definition  of  flexibility.  Distension,  as  defined  here,  is  the  concept  we  are 
interested  in  as  it  is  the  only  dimension  that  relates  to  new  capability.  Unfortunately, 
what  appears  to  be  missing  from  Fitzgerald’s  approach  is  how  exactly  to  establish  and 
value  distension,  which  is  the  crux  of  the  problem. 
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Baykasoglu  [2009]  uses  a  similar  three-dimensional  framework  to  quantify  flexibility, 
but  replaces  “distension”  with  probability.  To  Baykasoglu,  the  inclusion  of  probability  is 
vital,  and  is  overlooked  surprisingly  often  by  other  researchers.  After  all,  he  argues,  the 
value  of  flexibility  is  a  function  of  uncertainty,  and  thus  the  likelihood  that  a  given 
change  will  occur  must  be  taken  into  account.  Thus,  a  system  should  not  be  considered 
substantially  more  flexible  than  another  system  simply  because  it  can  more  easily 
accommodate  a  change  that  is  very  unlikely  to  occur.  Although  Baykasoglu  is  primarily 
concerned  with  manufacturing  flexibility,  he  believes  his  approach  has  general 
applicability,  “if  properly  implemented.” 

His  approach  involves  creating  a  matrix  that  captures  all  possible  states  of  a  particular 
system,  along  with  the  efficiency  of  each  state,  and  the  probability  of  switching  to  a 
different  state.  From  the  resulting  matrix,  a  value  known  as  the  “permanent”  is 
calculated.  The  permanent  is  similar  to  the  determinant,  except  that  only  absolute 
values  of  the  diagonal  products  are  used,  so  there  are  no  negative  components  in  the 
calculation.  Its  graph  theory  interpretation  is  the  sum  of  weights  of  perfect  matchings  in 
a  bipartite  graph.  Baykasoglu’s  approach  would  appear  valid,  but  his  explanation  of  the 
process  lacks  clarity.  In  addition,  it  seems  likely  that  populating  the  probability  and 
efficiency  aspects  of  the  model  would  be  difficult  in  practice. 

Flexibility  Dimension 

Cormier  [2008]  aims  to  provide  flexibility  metrics  for  use  in  the  early  stages  of  the 
system  design  process.  Operating  under  the  premise  that,  “the  flexibility  in  the  initial 
product  architecture  will  largely  determine  the  overall  flexibility  of  the  final  system,” 
Cormier  ultimately  wishes  to  provide  a  method  for  the  designer  to  select  the  most 
flexible  product  architecture.  The  author  proposes  to  measure  flexibility  along  three 
dimensions: 

•  Flows  between  subsystems 

•  The  connections  between  subsystems 

•  The  geometry  of  a  system 

The  subsystem  flows  are  intended  to  capture  the  range  of  possible  transfers  of  material, 
energy,  or  signal  out  of  the  subsystem  into  the  target  (i.e.,  design  space).  Then,  “the 
various  ranges  of  flow  characteristics  that  a  subsystem  can  handle  are  compared  to 
target  ranges  to  evaluate  the  flexibility  of  a  flow.”  At  this  time,  Cormier’s  model  assumes 
that  the  mapping  between  the  design  space  and  flexibility  is  linear,  but  hopes  to  address 
nonlinear  mappings  in  future  research.  The  connections  between  subsystems  are 
essentially  an  interface  analysis,  where  higher  numbers  of  interfaces,  and  interfaces  that 
do  not  provide  functionality  are  penalized  in  Cormier’s  scoring  methodology.  Finally, 
the  ability  of  the  system  to  expand  and  contract  to  accommodate  changes  in 
functionality  or  performance  is  considered.  Cormier  refers  to  this  concept  as  the  system 
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geometry,  and  involves  constructing  two  matrices  that  are  intended  to  capture  the 
system’s  expansion  and  contraction  characteristics. 

These  three  dimensions  then  serve  as  the  vertices  of  a  flexibility  cube  in  which  the  final 
flexibility  rating  is  determined  by  its  location  in  the  space  (a  score  of  zero  would  be  the 
most  inflexible  product  imaginable).  Unfortunately,  it  is  not  evident  what  value  this 
measure  is,  or  why  it  should  be  considered  valid. 

Flexibility  Index 

In  his  article  on  what  he  calls  “developmental  flexibility”  [Thomke,  1998],  proposes  the 
use  of  a  flexibility  index  to  measure  how  well  a  given  system  responds  to  a  particular 
change.  The  Flexibility  Index  is  calculated  by  dividing  the  percent  change  in  a  given 
attribute  by  the  percent  change  in  projected  profits.  So,  for  example,  if  it  costs  five 
percent  of  projected  profits  to  increase  the  battery  lifetime  of  a  product  by  twenty 
percent,  then  the  Flexibility  Index  of  this  product  for  this  attribute  is  twenty  divided  by 
five,  which  equals  four.  This  results  in  a  measure  of  the  economic  cost  of  modifying  a 
particular  product  feature. 

The  flexibility  index  does  provide  a  measure  of  investment  cost— at  least  in  terms  of  a 
ratio— which  is  good.  Of  course,  the  denominator  of  the  ratio  is  not  what  we’re  looking 
for,  as  profit  is  not  the  motivating  factor  in  defense  acquisition.  There  are  some  other 
obvious  concerns  in  using  this  approach  for  our  purposes.  First,  the  method  of 
calculating  the  flexibility  index,  while  logical,  is  not  justified  by  the  author  via  any 
scientific  or  mathematical  rationale.  Second,  this  technique  is  necessarily  attribute- 
specific,  thus  only  providing  an  indication  of  the  system’s  ability  to  change  with  respect 
to  a  single  feature,  not  an  overall  indication  of  the  system’s  ability  to  accommodate 
change.  To  gain  insight  into  the  whole  system,  a  series  of  flexibility  indices  would  need 
to  be  calculated  and  amalgamated.  Moreover,  this  approach  is  not  applicable  to  cases  of 
unforeseeable  sources  of  change,  which  is  also  a  drawback.  Finally,  and  most 
fundamentally,  the  flexibility  index  doesn’t  measure  any  inherent  characteristic  of  the 
product  at  all— it  simply  provides  an  indication  of  return  on  investment  to  facilitate  a 
single  attribute  decision  analysis. 

Flexible  Platform  Design  Process 

One  of  the  most  comprehensive  (though  not  entirely  lucid)  techniques  is  developed  by 
Suh  [2007].  He  describes  an  end-to-end  normative  design  process,  which  takes  into 
account  external  sources  of  uncertainty  to  achieve  flexible  systems.  The  seven-step 
process,  which  mostly  consists  of  organizing  and  collating  existing  methods,  is  called  the 
Flexible  Platform  Design  Process  (FPDP).  The  steps  are  as  follows: 

1.  Identity  market,  variants,  and  uncertainties 

2.  Determine  uncertainty-related  key  functional  attributes  and  design  variables 

Contract  Number:  H98230-08-D-01 71  DO  02,  TO  02  RT018 

Report  No.  SERC-2010-TR-010 
09/25/2010 

UNCLASSIFIED 

120 


UNCLASSIFIED 


3.  Optimize  product  family  and  platform  bandwidth 

4.  Identity  critical  elements  for  flexibility  (change  propagation  analysis,  engineering 
expertise) 

5.  Create  flexible  design  alternatives  (brainstorming,  concept  screening  and  scoring 
matrix) 

6.  Determine  costs  of  design  alternatives  (Parametric  cost  modeling) 

7.  Uncertainty  Analysis  (Decision  Trees,  NPV,  Real  Options) 

The  first  step  uses  clustering  analysis  and  conjoint  analysis.  The  second  step  can  be 
accomplished  via  QFD.  Step  three  is  done  via  gradient -based  optimization  and  heuristic 
optimization  (not  covered).  CPA,  or  “engineering  expertise”  is  used  in  the  fourth  step. 
Step  five  is  accomplished  via  brainstorming.  Parametric  cost  modeling  is  the 
recommended  method  for  step  six.  And  NPV  or  real  options  are  leveraged  at  the  final 
step. 

With  the  possible  exception  of  step  four,  Suh  appears  to  have  contributed  little  new 
content  to  the  topic  of  quantifying  the  value  of  flexibility.  Nevertheless,  Suh’s  holistic 
approach  is  welcome,  and  as  Suh  notes,  step  four  is  a  critical  aspect  of  the  problem.  Suh 
reasonably  argues  that  other  methods  jump  to  a  valuation  technique  prematurely, 
failing  to  explicitly  differentiate  all  the  potential  flexible  design  options  in  terms  of 
which  is  likely  to  provide  the  greatest  “bang  for  the  buck.” 

The  Formula  for  Flexibility 

Occasionally,  authors  have  provided  an  entirely  mathematical  conception  of  flexibility, 
thereby  automatically  allowing  for  quantification.  We  conclude  our  survey  by 
summarizing  two  such  approaches.  One  author  defines  flexibility  as  the  “partial 
derivative  of  the  total  instantaneous  cost  with  respect  to  a  given  state”  [Bordoloi,  1999]. 
Another  looks  to  identify  the  optimal  flexibility/cost-tradeoff  curve  of  a  system  using  the 
following  formula: 


/impl(7)  =  a+(7)  * 


y.x3/ 


V'C  7.^  £ -0 .  r 


/impl('T') 


—  (fy.v&l  —  1)  for  y.'T  f  0 
1  otherwise 


Which  he  describes  in  words  as— 

“The  flexibility  of  a  cluster ,  if  ever  activated,  is  calculated  by  the  sum  of  all  its 
interfaces’ flexibilities  minus  the  number  of  its  interfaces  less  1,  and  1  if  there  is  no 
interface  in  the  given  cluster.  The  flexibility  of  an  interface  is  the  sum  of  flexibilities  of 
all  its  associated  clusters”  [Haubelt,  2002] 
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While  the  purely  mathematical  approaches  to  quantifying  flexibility  are  intriguing,  it’s 
far  from  clear  that  either  conception  has  any  practical  merit.  The  first  idea  based  on 
“instantaneous  cost”  doesn’t  seem  relevant  to  design  flexibility  (though  it  claims  to),  nor 
does  the  above  formula.  In  fact,  the  proposed  formula  seems  instead  to  be  a  measure  of 
component  interdependence,  which  would  be  a  submeasure  of  modularity,  not 
flexibility. 

Summary 

The  model  for  quantifying  the  value  of  flexibility  needs  to  meet  multiple  criteria.  First, 
it  must  be  theoretically  valid.  The  model  we  seek  needs  to  be  capable  of  correctly 
measuring  the  degree  of  flexibility  in  a  system  design  (or  proposed  system  design)  and 
enabling  credible  value  decisions  in  the  face  of  uncertainty.  Second,  the  model  must  be 
demonstrably  valid.  It  must  predict  testable  results,  and  allow  for  verification  and 
validation  via  the  application  of  case  studies.  Third,  the  model  must  be  usable.  Its 
outputs  should  be  readily  understood  by  decision-makers,  and  it  should  have  the 
capability  to  be  readily  applied  by  practitioners,  to  include  having  data  inputs  that  can 
be  obtained  in  a  reasonably  straightforward  manner,  and  with  a  minimum  amount  of 
subjectivity.  Last,  our  ideal  flexibility  value  model  needs  to  be  applicable  to  the  DoD. 
Defense  acquisition  is  fundamentally  different  from  private-sector  acquisition  in  that 
system  value  is  not  ascertained  based  on  profit  forecasts;  rather,  value  is  determined  by 
a  system’s  capabilities,  and  how  well  it  meets  warfighter  needs. 

This  discussion  shows  that  there  are  many  general  and  specific  tools,  processes,  and 
methods  for  developing  such  a  model.  While  many  of  the  framework  tools  may  be 
useful  to  us,  none  of  the  fully-formed  methods  meet  all  of  our  criteria.  Many  are  of 
questionable  scientific  validity.  Several  others  may  be  valid,  but  fail  to  demonstrate 
their  validity,  or  are  too  esoteric  to  allow  for  any  possibility  of  genuine  validation.  Some 
of  the  most  promising  models  are  the  most  difficult  to  understand  and  most  difficult  to 
actuate  (i.e.,  filtered  outdegree,  flexibility  aspects,  and  flexible  platform  design  process). 
And  only  a  couple  of  techniques  are  well  suited  to  the  alternate  value  strategies  of  the 
military,  while  none  included  a  defense-based  case  study.  Most  problematic  of  all,  far 
too  many  of  the  approaches  take  aim  at  the  wrong  concept  entirely,  seeking  to  measure 
flexibility,  per  se,  rather  than  the  value  of  flexibility. 


A.3:  MPTs  for  Incorporating  Flexibility  in  Systems 

Following  the  overview  of  the  different  definitions  of  system  flexibility,  its  measure,  and 
the  valuation  of  flexibility,  this  section  focuses  on  the  available  methods,  processes  and 
tools  to  incorporate  flexibility  into  system  design  or  allow  for  fielded  systems  to  be 
adapted  to  changing  environment  characteristics. 
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There  are  also  several  classes  of  MPTs  for  improving  flexibility.  These  include  modular 
and  service-oriented  architectures;  domain  ontologies;  interoperability  connectors; 
autonomy  and  adaptive  control;  agile  methods;  concurrent  engineering;  robust 
optimization;  delayed  differentiation  and  user  programmability.  The  following  sections 
discuss  some  of  these  approaches  in  detail. 

Modularity 

A  frequent— though  perhaps  overly  simplistic— recommendation  for  achieving  flexibility 
is  to  simply  implement  modularity.  A  number  of  studies  come  to  this  conclusion  based 
on  empirical  approaches. 

In  response  to  our  sponsors’  priorities,  our  primary  research  focus  will  be  in  valuing  the 
flexibility  to  adapt  to  foreseeable  sources  of  change.  For  this  case,  a  very  powerful 
strategy  is  to  modularize  the  system’s  architecture  around  these  sources  of  change.  For 
software,  a  strong  theoretical  and  practical  basis  for  this  strategy  was  developed  in 
[Parnas,  1979].  It  has  subsequently  been  found  to  work  well  for  systems  including 
hardware,  software,  and  human  factors.  If  this  is  done,  when  the  foreseeable  changes 
come,  their  adaptation  effects  are  confined  to  single  module. 

Acting  on  the  presumption  that  “flexibility  is  an  existing  property  of  certain  products 
which  results  from  design  choices  made  by  their  inventors,”  two  research  projects  chose 
to  use  the  U.S.  patent  database  as  a  means  of  collecting  data  regarding  design  choices 
for  various  inventions.  Qureshi  [2006]  was  the  first,  identifying  and  analyzing  90 
patents.  Based  on  his  analysis,  he  arrived  at  a  rather  unwieldy  set  of  seventeen  new 
principles  of  flexibility,  which  he  then  binned  (mercifully)  into  four  broad  principles: 

•  Increase  the  degree  of  modularity  of  a  device:  This  recommendation  included 
obvious  suggestions  like  “using  a  different  module  to  carry  out  each  different 
function”  and  “dividing  each  module  into  a  number  of  smaller,  identical 
modules” 

•  Reduce  the  communications  between  modules  and  enable  the  device  to  function 
normally  regardless  of  the  orientation,  location  and  arrangement  of  its 
individual  modules 

•  Facilitate  the  addition  of  new  functionality:  This  included  the  recommendation 
to  locate  “those  parts  which  are  anticipated  to  change  near  the  exterior  of  the 
device  and  those  which  are  not  near  its  center” 

•  Enable  the  device  to  respond  to  minor  changes 

There’s  really  not  a  lot  to  be  learned  by  these  recommendations  on  how  to  make  a 
product  more  flexible.  The  first  two  suggestions  can  be  summarized  as,  “make  the 
product  more  modular,”  and  the  last  two  are  tautological,  essentially  saying,  “make  the 
product  flexible.” 
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Keese  [2007]  expanded  on  Qureshi’s  patent  work  by  merging  it  with  an  empirical  study 
of  various  consumer  products  that  were  also  deemed  to  be  flexible.  Interestingly,  he 
uses  CMEA  to  identify  flexible  design  aspects.  He  arrives  at  a  very  similar  set  of 
recommendations  as  those  of  Qureshi,  which,  again,  can  be  boiled  down  to  “make  the 
product  modular  and  make  the  product  flexible.” 

•  Increase  the  degree  of  modularity  of  a  device 

•  Reduce  the  number  of  parts  requiring  manufacturing  changes 

•  Reduce  the  communication  between  modules ,  and  enable  the  device  to  function 
normally  regardless  of  the  orientation,  location  and  arrangement  of  its 
individual  modules 

•  Facilitate  the  addition  of  new  functionality  and  rearrangement  or  scaling  of 
parts 

•  Enable  the  device  to  respond  to  minor  changes 

Aside  from  the  lack  of  novelty  in  the  recommendations  for  both  of  these  studies,  the 
obvious  concern  is  whether  the  screening  criteria  for  identifying  flexible  products  were 
valid.  For  all  of  the  data  in  Qureshi’s  method,  as  well  as  half  of  the  data  in  Keese’s 
approach,  candidate  flexible  products  were  identified  via  a  database  search  (i.e.,  the  U.S. 
patent  database).  This  search  entailed  criteria  like,  “references  to  other  evolutions, 
multiple  preferred  embodiments,  and  direct  references  to  design  flexibility  in  their  text.” 
It’s  not  apparent  that  these  search  terms  would  yield  truly  flexible  products.  In  terms  of 
Keese’s  adjunct  product  listing,  since  it  relied  on  CMEA,  the  same  concerns  raised 
previously  regarding  its  validity  would  apply. 

Rajan  [2005]  also  used  empirical  data  to  evaluate  product  flexibility,  and  also  employed 
CMEA.  As  part  of  his  findings,  Rajan  provides  guidelines  in  order  to  aid  in  designing 
flexibility.  In  Rajan  [2003],  he  offered  the  following  recommendations  to  achieve 
flexibility: 

•  Improve  the  design  flexibility  by  making  the  device  more  modular 

•  Reduce  the  effect  of  a  change  in  a  design  by  increasing  the  number  of  partitions 

•  Reduce  the  effect  of  a  change  by  increasing  the  number  or  size  of  buffer  zones 

•  Reduce  the  occurrence  of  changes  by  standardizing  components  and  interfaces 

•  Reduce  the  occurrence  of  a  change  by  increasing  the  performance  envelope 

•  Reduce  occurrence  of  changes  by  selecting  technology  which  is  far  from 
obsolescence 

The  first  four  recommendations  are  related  to  modularity.  The  fifth  is  more  about 
versatility  than  flexibility.  And  the  last  may  be  a  reasonable  systems  engineering 
heuristic  under  certain  conditions  (e.g.,  when  mission  life  duration  is  a  priority),  but  it 
doesn’t  seem  to  directly  allow  for  more  flexibility.  If  anything,  the  direction  of  causality 
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would  seem  to  flow  the  other  way,  i.e.,  a  more  flexible  design  will  lead  to  the  use  of 
technologies  further  from  obsolescence. 

In  Raj an  [2005],  he  amplifies  on  these  recommendations  in  the  form  of  several 
additional  guidelines: 

•  Modularizing  the  design  leads  to  more  product  flexibility.  As  the  design 
becomes  more  integrated ,  it  becomes  more  inflexible  for  redesign. 

•  “Designing  the  modules  in  a  product,  as  external  attachments,  makes  the  design 
even  more  flexible. 

•  Designing  with  more  standard  components  and  interfaces  will  improve  product 
flexibility. 

•  Directed  partitioning  of  a  design  into  a  greater  number  of  elements  (manifested 
through  higher  numbers  of  components  and  functions)  improves  the  flexibility . 

•  Reducing  the  number  of  parts  within  modules,  after  effective  layout,  does  not 
affect  flexibility  (this  insight  must  be  verified  in  future  studies).  The  implication 
is  to  have  simultaneous  design  for  improved  assemblability,  while  maintaining 
flexibility” 

Aside  from  our  perfectly  justifiable  desire  to  throttle  Raj  an  for  introducing  the  most 
ridiculous  ility”  yet  (i.e.,  “assemblability”),  we  find  that  the  result  is  again  just  a  list  of 
heuristics  to  achieve  modularity.  Rajan  could  have  made  his  point  more  succinctly  by 
simply  saying,  “If  you  want  to  implement  flexibility,  make  sure  you  implement 
modularity.” 

Design  For  Adaptability 

Another  approach  to  implement  flexibility  is  through  a  method  known  as  Design  for 
Adaptability,  or  DFAD.  While  certain  articles  on  DFAD  neglect  to  define  what  they 
mean  by  adaptability  [Lipson,  2001],  it  is  apparent  from  context  that  the  principle 
involved  is  close  enough  to  flexibility  to  be  of  potential  interest  to  us.  The  driving 
principle  of  DFAD  is  to  design  the  product  so  that  it  can  have  a  longer  useful  life.  From 
Kasarda  [2007]— 

“The  DFAD  methodology  concept  is  based  on  the  hypothesis  that  product  life  ends 
because  a  product  is  unable  to  adapt  to  change.  A  product  may  be  retired  for  myriad 
reasons  including  that  it  is  broken,  out  of  style,  or  has  become  inefficient  due  to 
technology  obsolescence.  In  these  cases,  the  product  was  not  able  to  adapt  to  change— 
it  was  unable  to  self-heal,  it  could  not  modify  or  reconfigure  to  meet  changing  fashion 
needs,  or  it  could  not  be  upgraded,  for  physical  or  economic  reasons,  to  utilize  new 
technology.  To  address  these  and  similar  issues,  we  are  developing  the  DFAD 
methodology.” 
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DFAD  is  described  as  being  similar  to  the  Darwinian  process  of  evolution  and 
adaptation  [Kasarda,  2007].  More  formally,  it  is  rooted  in  control  theory  in  that 
products  are  modeled  as  dynamic  systems  with  feedback  control  mechanisms  for 
adapting  to  “changes  in  product  performance  criteria”  in  order  to  achieve  a  longer  useful 
life  [Kasarda,  2007;Lipson,  2001]. 

Unfortunately,  little  else  can  be  said  regarding  DFAD.  The  research  stream  appears  to 
have  withered,  as  there  are  few  publications  that  refer  to  it,  and  the  actual  usable 
content  in  the  publications  is  sparse.  Without  more  details  on  its  application,  DFAD 
does  not  appear  to  be  a  viable  implementation  strategy. 

Design  For  Changeability 

Schulz  [1999]  creates  a  method  to  implement  flexibility  that  he  calls  Design  for 
Changeability  (DFC).  DFC  is  comprised  of  four  strategic  attributes:  flexibility,  agility, 
robustness,  and  adaptability.  Schulz  refers  to  systems  as  “intelligent”  if  they  consist  of 
all  four  attributes.  He  then  introduced  terminology  related  to  “Basic  Principles”  and 
“Extending  Principles.”  The  Basic  Principles  support  all  four  attributes  and  consist  of 
Ideality/Simplicity,  Independence,  and  Modularity,  Encapsulation  (which  he  defines  via 
DSMs).  The  extending  principles  have  more  restrictive  application  (i.e.,  don’t  apply  to 
all  four  strategic  attributes),  and  include  a  number  of  principles  such  as  Integrability, 
Autonomy,  Scalability,  Decentralization,  Redundancy,  and  Reliability. 

Unfortunately,  the  author’s  discussion  ends  abruptly  with  this  outline  of  design 
principles,  without  any  case  study  examples  or  discussion  of  how  to  use  these  principles 
effectively  to  achieve  genuine  changeability  in  practice.  This  is  a  key  step  as  the 
principles,  as  delineated  by  Schulz,  are  compelling,  but  rather  abstract.  The  author 
recognizes  these  shortcomings  and  states  that  he  plans  to  address  them  in  future 
research;  however,  that  did  not  appear  to  happen  as  the  DFC  concept  has  not  appeared 
in  the  literature  subsequently. 

Acquisition  Strategy 

Presumably,  we  can  cultivate  flexibility  in  the  design  process  to  some  degree  via  the 
specific  acquisition  strategies  we  choose  to  employ.  For  instance,  consider  the  contract 
type.  If  we  elect  to  use  a  firm-fixed  price  contract,  we  gain  (in  theory)  a  guaranteed 
capability  at  a  guaranteed  price.  However,  we  sacrifice  the  ability  to  quickly  respond  to 
new  information  or  priorities  that  may  emerge  as  we  progress  through  the  program  as 
we  might  do  under  a  level-of-effort  (i.e.,  cost-plus)  contract.  This  is  also  why  it  is  not  a 
good  idea,  in  general,  to  procure  a  system  using  a  firm  fixed  price  contract  if  there  is 
sizeable  uncertainty.  The  contract  type  is  one  element  of  our  acquisition  strategy  that  is 
likely  to  have  a  bearing  on  our  ability  to  develop  a  flexible  system. 
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Although  there  does  not  appear  to  be  any  research  on  how  system  flexibility  relates  to 
contract  type,  there  is  some  discussion  in  the  literature  regarding  how  another  aspect  of 
acquisition  strategy  may  foster  flexibility.  Recall  that  in  a  previous  section,  we  noted  that 
flexibility  only  had  value  under  conditions  of  uncertainty,  and  touched  on  the  fact  that 
keeping  options  open  as  long  as  possible  was  a  method  of  mitigating  the  impacts  of 
uncertainty.  Delaying  commitment  (and  thus  remaining  flexible  longer)  can  also  be 
accomplished  via  certain  procurement  strategies,  namely  incremental  development  and 
iterative  development.  Mikkonen  [2001],  in  discussing  software  flexibility,  describes 
incremental  development  as  a  technique  to  “leave  some  parts  of  the  architecture  open,” 
and  “provide  stub  functions  for  unimplemented  features,”  the  exact  capabilities  that 
we’re  looking  for  with  design  flexibility. 

Iterative  development  provides  flexibility  in  a  similar  manner,  as  design  commitments 
occur  in  steps  thereby  allowing  for  the  accrual  of  time  and  knowledge  to  provide  more 
flexibility  with  respect  to  design  decisions:  “Iterative  development  requires  accepting 
the  fact  that  requirements  cannot  always  be  adequately  specified  without  studying 
systems  iteratively,”  and  that  rapid  incremental  prototyping  is  essential  to  obtaining 
new  system  insights  and  improved  implementations  [Mikkonen,  2001].  Along  these 
lines,  some  have  suggested  that  prototyping  a  system  is  actually  superior,  in  most  cases, 
to  the  practice  of  specifying.  According  to  Boehm  [1984],  prototyping  allows  “changes 
late  in  the  design  process  as  a  result  of  new  information  from  customers  resulted  in 
products  that  were  not  only  judged  superior  from  a  customer  perspective  but  also 
developed  with  fewer  design  resources.” 

According  to  some,  another  acquisition  strategy  for  achieving  flexibility  is  to  simply 
provide  for  reserve  margins  in  all  performance  areas  where  changes  are  deemed  more 
likely  to  impact  the  system  design  [Eckert,  2004;  Krishnan,  2002].  Note  that  this  is 
somewhat  distinct  from  the  over-capacitization  approach  discussed  earlier  when 
distinguishing  flexibility  from  versatility.  Now,  we  are  referring  to  over-designing  the 
system  so  that  perturbations  are  less  likely  to  result  in  a  reduction  of  performance  below 
a  required  threshold.  What  this  would  mean  in  practice  is  investing  in  a  design 
approach  that  could  seamlessly  accommodate  (vice  allow  for  modification,  under  the 
stricter  view  of  flexibility)  various  uncertain  alternatives.  The  hope  is  that  the 
investment  is  worthwhile,  as  the  “potentially  expensive  redundancy  is  designed  into  the 
product,  making  future  redesign  much  cheaper”  [Eckert,  2004].  Note,  however,  that  at 
the  moment  that  the  reserve  margin  is  implemented,  uncertainty  (for  that  parameter) 
disappears,  and  flexibility  is  no  longer  germane. 

Krishnan  [2002]  presents  a  related  strategy  to  achieve  the  same  effect.  By  committing 
to  one  or  more  “parallel  project  paths,”  the  program  has  more  flexibility  to  contend  with 
multiple  source  of  uncertainty.  Note,  however,  that  this  strategy  is  only  applicable  to 
foreseeable-type  changes.  It  should  also  be  evident  that  this  discussion  is  closely  related 
to  basic  principles  of  risk  management,  and  serves  to  elucidate  the  intricate 
relationships  among  flexibility,  uncertainty,  risk,  and  opportunity. 
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In  broader  program  management  terms,  design  flexibility  may  also  encompass, 
“capability  restoration,  capability  augmentation,  risk  diversification,  schedule 
diversification,  or  uncoupling  of  system  requirements”  [Brown,  2007].  Ross  [2008] 
calls  for  acquisition  policy  changes  that  would  allow  unused  change  mechanisms  to  be 
tracked  and  managed  at  lower  levels,  similar  to  the  concept  of  management  reserve,  and 
thus  could  be  thought  of  as  “change  reserve.”  Ross  references  Boeing’s  development  of 
the  JDAM,  and  notes  that  this  type  of  approach  was  employed  successfully.  By  allowing 
the  contractor  to  maintain  ownership  of  the  design,  it  “facilitated  the  changing  of  the 
design  over  time  with  less  ‘cost.’”  Given  the  authority  to  manage  the  change  reserve, 
Boeing’s  design  was  highly  modular,  COTS-intensive,  and  consisted  of  interfaces  with 
excess  capacity. 

Other  Strategies 

We  will  close  this  chapter  on  implementing  flexibility  with  a  quick  look  at  some  other 
methods  discussed  in  the  literature  that,  while  interesting,  are  not  substantive  enough  to 
warrant  detailed  consideration. 

According  to  Willems,  et  al.  [2003],  quantifying  a  product's  adaptability  can  be  achieved 
through  a  process  they  have  named  Methodology  for  Assessing  the  Adaptability  of 
Products  (MAAP).  This  process  purportedly  supports  the  identification  of  improvement 
potential  in  the  design  of  the  product,  and  its  components,  though  the  applicability 
requires  that  the  types  of  changes  that  must  be  adapted  to  be  known  a  priori.  The 
methodology  is  validated  via  a  case  study  involving  cellular  phones. 

The  principle  of  open  architecture  is  another  possible  approach  to  achieving  flexibility. 
Increasingly  effective  in  the  software  domain,  the  idea  has  been  suggested  that  this  type 
of  architectural  approach  could  be  extended  to  hardware  [Piller,  2010].  The  idea  is  that 
“embedded  toolkits”  could  be  provided  to  designers,  allowing  them  to  “design  products 
with  build-in  [sic]  flexibility  by  embedding  knowledge  and  rules  about  possible  product 
differentiations  into  the  product.”  This  is  a  very  recent  article,  and  the  concept  is  not  yet 
mature;  at  this  time,  the  authors  are  looking  for  a  “proof  of  concept”  opportunity. 

Summary 

“It  is  not  possible  to  know  exactly  how  a  particular  design  will  perform  until  it  is  built. 
But  the  product  cannot  be  built  until  the  design  is  selected.  Thus,  design  is  always  a 
matter  of  decision  making  under  conditions  of  uncertainty  and  risk”  [Hazelrigg,  1998]. 

Hazelrigg’s  observation  goes  to  the  heart  of  the  problem  in  implementing  flexibility.  We 
know  that  it  is  incumbent  upon  the  procuring  agency,  not  the  user,  to  infuse  flexibility 
into  the  system.  This  is  undoubtedly  a  challenging  endeavor  as  the  literature  is  littered 
with  many  porous  attempts.  As  Hazelrigg  observed  over  a  decade  ago  when  referring  to 
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coeval  design  approaches:  “these  methods  are  ad  hoc  approaches  that  are  not  rooted  in 
any  fundamental  theory,  nor  do  they  provide  a  basis  for  engineering  design  as  a 
discipline.”  Unfortunately,  it  appears  that  little  has  changed  since  then.  Based  on  what 
we  can  glean  from  the  literature,  perhaps  the  only  lesson  is  that  if  we  can  implement 
modularity,  then  we  have  gone  a  long  way  in  implementing  flexibility.  Though,  even  for 
this,  the  evidence  is  more  anecdotal  than  analytical. 
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Appendix  B:  Case  studies 


B.1:SHIPMAIN  Case  Study 


Data  Collection:  Aggregate  data  was  gathered  during  an  initial  KVA  knowledge  audit 
conducted  via  survey  and  a  group  interview  setting  at  NAVSEA,  Washington  Navy  Yard, 
DC.  Three  SHIPMAIN  SMEs  were  present  at  the  group  interview,  and  each  had 
expertise  related  to  the  SHIPMAIN  process.  The  three  SMEs  each  have  over  30  years 
experience  in  the  shipyard  industry,  with  a  high  degree  of  expertise  in  their  affiliated 
disciplines.  Their  input  will  be  statistically  analyzed  for  reliability,  and  all  estimates  will 
be  aggregated  to  reflect  the  cost  and  number  of  process  executions  averaged  over  five 
years.  Business  rules  for  Phases  IV  and  V  of  the  SHIPMAIN  process  guided  the 
interview. 

Phases  IV  and  V  of  the  SHIPMAIN  process  were  created  from  input  and  discussion  by 
various  stakeholders  at  NAVSEA,  Type  Commanders  (TYCOM),  public  and  private 
shipyards,  Space  and  Naval  Warfare  Systems  Command  (SPAWAR),  Office  of  the  Chief 
of  Naval  Operations  (OPNAV)  and  other  entities  with  a  vested  interest  in  maintenance 
and  modernization  efforts  (Commander,  Naval  Sea  Systems  Command,  2006). 
Business  rules  for  these  phases  are  regularly  reviewed  and  updated  to  be  properly 
aligned  with  business  goals  and  the  needs  of  Fleet  Commanders.  Currently,  Phases  IV 
and  V  of  SHIPMAIN  are  not  in  a  functionally  implemented  state  but  are  rather  in  an 
early  adoption  period  while  business  rules/processes  mature  and  long-standing  legacy 
practices  give  way  to  the  SHIPMAIN  process.  A  key  assumption  of  this  proof-of-concept 
case  is  that  the  SHIPMAIN  process  functions  as  described  in  the  business  rules  listed  in 
Appendix  D  of  the  SSCEPM  dated  December  11,  2006. 

Methodology:  The  method  of  analysis  for  this  proof  of  concept  is  the  Learning  Time 
method. 3  A  thorough  discussion  and  review  of  current  SHIPMAIN  business  rules  with 
the  SMEs  established  what  processes  constitute  the  core  of  SHIPMAIN  Phases  IV  and  V, 
identified  the  inputs  and  outputs  of  those  processes,  and  determined  the  frequency  of 
core  process  iterations.  The  discussion  further  established  boundaries  between  the 
defined  processes  in  order  to  effectively  apply  the  KVA  methodology  and  to  properly 
identify  and  valuate  the  knowledge  required  for  each.  Eight  core  processes  were 
identified,  and  detailed  descriptions  of  each  were  provided  by  the  SMEs  and  the 
SHIPMAIN  business  rules.  Each  core  process  requires  a  certain  level  of  knowledge  in 
one  or  more  of  the  following  areas:  administration,  management,  scheduling,  budgeting, 
basic  computer  skills,  engineering,  shipboard  systems,  logistics  or  project  management. 


3  See  Appendix  A  for  a  detailed  discussion  of  Learning  Time. 
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The  SMEs  spent  considerable  time  contemplating  the  amount  of  knowledge  embedded 
in  each  core  process,  and  provided  ALT  estimates  for  each.  The  established  baseline 
level  of  knowledge  for  consideration  was  a  GS-13  employee  with  1  year  of  experience  and 
a  college  degree  (no  field  specified).  Finally,  the  team  of  SMEs  provided  individual  and 
uninfluenced  RLT  and  rank-order  estimates,  which  lead  to  a  correlation  of  greater  than 
80  percent— thereby  establishing  a  high  level  of  reliability  on  the  ALT  figures  obtained. 
Additional  discussion  occurred  spontaneously  among  the  SMEs,  which  lead  to  a  group 
conclusion  that  Blocks  265  and  300  were  equivalent  in  complexity.  Adjusting  the  RLT 
and  rank  order  to  reflect  that  conclusion  leads  to  greater  than  a  90-percent  correlation 
across  the  data  fields. 

Key  Assumptions:  As  previously  mentioned,  this  analysis  is  based  on  information 
collected  from  previous  research  by  LT  Christine  Komoroski  (2005),  SMEs  from 
NAVSEA,  data  contained  in  the  NDE  and  current  directives.  For  the  purposes  of  this 
study,  all  maintenance  and  modernization  efforts  are  assumed  to  occur  as  described  in 
the  current  business  rules  listed  in  Appendix  D  of  the  SSCEPM  dated  December  11, 
2006.  It  is  also  important  to  keep  in  mind  that  maintenance  and  modernization  efforts 
vary  substantially  in  number,  manpower  requirements,  duration  and  complexity.  After 
conducting  extensive  interviews  with  SMEs  and  conducting  a  thorough  review  of 
current  directives,  related  research  and  existing  data  in  the  NDE,  the  researchers  made 
the  following  assumptions: 

•  Of  1,200  annual  modernization  and  maintenance  availability  periods,  25  percent 
involve  low  complexity  installations,  25  percent  high  complexity  installations, 
and  50  percent  involve  medium  complexity  installations.  Assume  all  efforts  in 
this  study  involve  efforts  of  medium  complexity. 

•  On  average,  20  SCDs  are  generated  per  week. 

•  The  market  comparable  labor  rate  is  35  percent  greater  than  the  government 
labor  rate. 

•  Price  per  common  unit  of  output  is  $75.45. 

Discussion  of  As-is  Scenario 

Number  of  Employees.  The  number  of  employees  value  used  to  build  this  model 
represents  the  number  of  employees  assigned  to  complete  the  given  process  for  each 
cycle  or  iteration.  Numbers  assigned  are  based  on  interviews  with  SMEs.  By  accounting 
for  the  number  of  personnel  involved  in  each  process,  the  researchers  can  determine 
how  often  knowledge  is  used.  This  method  also  provides  an  approximate  way  to  weight 
the  cost  of  using  knowledge  in  each  process. 

Times  Performed  in  a  Year.  Estimations  for  the  number  of  times  each  process  is 
executed  per  year  are  based  on  the  aggregated  number  of  occurrences  for  each  process. 
The  NDE  was  queried  with  the  following  filters  to  gather  the  raw  data: 
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The  search  was  limited  to  title  “K”  and  “P”  alterations. 

•  FY  2002  through  2007. 

•  Ships  of  the  following  TYCOMs: 

o  Commander,  Naval  Air  Force  Atlantic 
o  Commander,  Naval  Air  Force  Pacific 
o  Commander,  Naval  Surface  Force  Atlantic 
o  Commander,  Naval  Surface  Force  Pacific 

These  filters  were  put  in  place  to  establish  a  five-year  average  of  maintenance  or 
modernization  availability  periods  for  all  surface  combatant  ships  to  include  Aircraft 
Carriers.  The  result  of  the  query  was  that  an  average  of  1,200  availability  periods  occur 
each  year.  This  number  was  conditionally  modified  to  take  the  complexity  of  installs 
during  availability  periods  into  consideration.  To  provide  a  reasonable  scope,  25  percent 
of  availability  periods  were  considered  to  be  simple,  25  percent  complex  and  50  percent 
moderate.  600  moderately  complex  installations  frame  the  scope  of  this  model. 

The  number  of  times  the  process  is  performed  for  the  remaining  blocks  is  based  on  the 
number  of  installations  that  occur.  For  each  installation  that  occurs,  a  SCD  is  generated, 
and  the  number  of  SCDs  provides  a  reliable  proxy  for  the  number  of  installations.  SMEs 
provided  data  and  analysis  which  estimates  an  average  of  20  SCDs  are  initiated  per 
week,  leading  to  1,040  SCDs  generated  annually.  Applying  the  same  conditional 
modifier  to  account  for  complexity,  520  SCDs  or  installs,  would  occur  each  year. 

Actual  Learning  Time.  In  order  to  determine  the  ALT  from  a  common  point  of 
reference,  the  SMEs  were  instructed  to  imagine  a  baseline  individual  of  a  college 
graduate  at  the  GS-13  civilian  rank  level  with  a  year  of  experience  in  some  sector  of  the 
shipyard  industry.  All  experts  understood  that  each  process  learning  time  estimate 
must  adhere  to  the  basic  assumptions  that  knowledge  is  only  counted  if  in  use,  and  the 
most  succinct  path  to  achieve  a  unit  of  output  must  be  considered.  Each  core  process 
was  broken  down  into  its  component  sub-processes,  and  respective  ALT  values  were 
assigned  for  each  sub-process.  The  final  ALT  value  for  each  core  process  is  a  summation 
of  the  sub-process  ALT  estimates.  Finally,  all  ALT  values  are  based  on  the  following 
time  assumptions: 

•  One  year  =  230  work  days 

•  One  month  =  20  work  days 

•  One  week  =  5  work  days 

•  One  day  =  8  hours 

Determining  Value.  Each  process  contains  a  certain  amount  of  process  automation- 
ranging  from  zero  to  100  percent.  The  amount  of  automation  is  a  proxy  for  how  much 
knowledge  is  embedded  in  the  IT  supporting  the  automation.  It  is  important  to  estimate 
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how  much  of  each  process  is  automated,  and  to  be  consistent  in  those  estimates,  so  that 
the  knowledge  embedded  in  the  technology  resources  is  accounted  for.  Upon 
determination  of  the  percentage  estimate,  the  Total  Learning  Time  (TLT)  is  calculated 
by  dividing  ALT  by  the  percentage  of  process  automation  for  that  process. 

The  TLT  value  is  then  multiplied  by  the  number  of  employees  and  the  number  of  times 
the  process  is  performed  per  year  to  establish  a  Total  Knowledge  factor.  The  Total 
Knowledge  factor  is  then  multiplied  by  a  price  per  common  unit,  based  on  market 
comparables,  to  derive  the  “benefits”  or  “value”  of  each  process.  The  resulting  product 
is  then  used  as  the  numerator  for  determining  ROK  and  ROI. 

Cost-estimation.  To  estimate  the  cost  of  government  employees  involved  in  the 
processes,  the  2007  civilian  pay  chart  was  referenced.  Each  civilian  pay  grade  has 
associated  “steps”  to  account  for  various  unique  factors  of  each  job.  All  pay  estimates 
are  based  on  Step  Six  of  the  associated  pay  grade.  Since  the  processes  take  place  across 
the  globe,  no  locality  pay  differentials  were  taken  into  consideration  to  minimize 
variation.  Also,  because  basic  computing  hardware  and  software  is  utilized  in  every 
scenario,  IT  cost  is  not  included  in  the  As-is  analysis.  It  is  assumed  that  each  employee 
in  this  process  has  an  email  account,  laptop  or  desktop  computer  with  identical  software 
and  has  access  to  a  printer.  Material,  travel,  and  other  miscellaneous  costs  are  not 
included  in  this  analysis  so  labor  cost  may  be  isolated. 

Establishing  a  market  comparable  for  government  labor  was  accomplished  by 
comparing  the  pay  of  contractors  who  conduct  the  same  type  and  scope  of  work  as  the 
government  employee.  The  contracted  base  pay  was  on  average  35  percent  higher  than 
the  government  employees.  Benefits,  locality  pay  differential  and  other  variables  were 
not  compared  to  establish  this  rate;  only  base  pay  was  considered.  All  government 
employee  rates  were  increased  by  35  percent  to  achieve  the  values  for  the  market  price 
used  to  establish  a  price  per  common  unit  of  output. 

Block  As-is  KVA  Data 


Block  265 

Hull  Installation  and  Risk  Assessment 

Sub  process 

Hourly 

Personnel 

Cost 

Head 

count 

Times  Pert. 

Per  Year 

Time  to 
Complete 
(Hrs) 

Annual 

Personnel  Cost 

%IT 

ALT  (Hrs) 

Total 

Knowledge 

Total  Benefits 

Annual  Cost 

ROK 

ROI 

265.1 

Installation  Procurement,  Design  & 
Advance  Planning 

$43.10 

35 

520 

160 

$125,507,200 

25% 

40 

970667 

$72,071,847 

$125,507,200 

57% 

43% 

265.2 

Hull  Installation  Readiness  Review 

$29.78 

2 

520 

40 

$1,238,848 

80% 

40 

208000 

$15,443,967 

$1,238,848 

1247% 

1147% 

265.3 

Evaluate  Maturity  Status 

$50.16 

1 

520 

20 

$521,664 

0% 

40 

20800 

$1,544,397 

$521,664 

296% 

196% 

265.4 

Provide  Risk  Assessment 

$50.16 

1 

520 

40 

$1,043,328 

0% 

56 

29120 

$2,162,155 

$1,043,328 

207% 

107% 

265.4.1 

Formally  Propose  Install  for 

Readniess  Assessment  and  Auth. 

$50.16 

1 

520 

20 

$521,664 

0% 

40 

20800 

$1,544,397 

$521,664 

296% 

196% 

265.5 

Risk/Readiness  Determination 

$59.01 

4 

130 

40 

$1,227,408 

0% 

56 

29120 

$2,162,155 

$1,227,408 

T7B 

76% 

Process  Totals: 

$94,928,918 

$130,060,112 

73% 

-27% 
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Block  270 

Authorize  Installation 

Sub  process 

Hourly 

Personnel 

Cost 

Head 

count 

Times  Pert. 
Per  Year 

Time  to 
Complete 
(Hrs) 

Annual 

Personnel  Cost 

%IT 

ALT  (Hrs) 

Total 

Knowledge 

Total  Benefits 

Annual  Cost 

ROK 

ROI 

270 

Installation  decision 

$76.00 

4 

520 

20 

$3,161,600 

85% 

24 

332800 

$24,710,347 

$3,161,600 

782% 

682% 

Resolvf 

Bk 

"Not  Aut 

)ck  280 
horized/Def 

erred  S 

Sub  process 

Hourly 

Personnel 

Cost 

Head 

count 

Times  Pert. 
Per  Year 

Time  to 
Complete 
(Hrs) 

Annual 

Personnel  Cost 

%IT 

ALT  (Hrs) 

Total 

Knowledge 

Total  Benefits 

Annual  Cost 

ROK 

ROI 

280 

Update  HMP.LOA  and  Fielding  Plan 

$29.78 

1 

520 

40 

$619,424 

75% 

24 

49920 

$3,706,552 

$619,424 

598% 

498% 

Blc 

Ins 

)ck  300 
tall  SC 

Sub  process 

Hourly 

Personnel 

Cost 

Head 

count 

Times  Pert. 

Per  Year 

Time  to 
Complete 
(Hrs) 

Annual 

Personnel  Cost 

°/dT 

ALT  (Hrs) 

Total 

Knowledge 

Total  Benefits 

Annual  Cost 

ROK 

ROI 

300 

Complete  installation  and  testing 

$42.45 

46 

520 

40 

$40,616,160 

25% 

40 

1275733 

$94,722,998 

$40,616,160 

233% 

133% 

Fee 

dback:  C 

Blc 

OSt,  CM,  F 

jck  310 
erformance 

Sched 

ule, ILS 

Sub  process 

Hourly 

Personnel 

Cost 

Head 

count 

Times  Pert. 

Per  Year 

Time  to 
Complete 
(Hrs) 

Annual 

Personnel  Cost 

°/dT 

ALT  (Hrs) 

Total 

Knowledge 

Total  Benefits 

Annual  Cost 

ROK 

ROI 

310 

Provide  Feedback  Data 

$29.78 

2 

520 

20 

$619,424 

0% 

24 

24960 

$1,853,276 

$619,424 

299% 

199% 

Bl 

Contir 

DCk  320 
tue  Installs 

Sub  process 

Hourly 

Personnel 

Cost 

Head 

count 

Times  Pert. 

Per  Year 

Time  to 
Complete 
(Hrs) 

Annual 

Personnel  Cost 

°/dT 

ALT  (Hrs) 

Total 

Knowledge 

Total  Benefits 

Annual  Cost 

ROK 

ROI 

320 

Determine  impact  on  future  installs 

from  Feedback  in  310 

$59.01 

5 

520 

20 

$3,068,520 

0% 

24 

62400 

$4,633,190 

$3,068,520 

151% 

51% 

Block  330 

Final  Install,  Closeout  SC 

Sub  process 

Hourly 

Personnel 

Cost 

Head 

count 

Times  Perf. 
Per  Year 

Time  to 
Complete 
(Hrs) 

Annual 

Personnel  Cost 

°/dT 

ALT  (Hrs) 

Total 

Knowledge 

Total  Benefits 

Annual  Cost 

ROK 

ROI 

330 

Verify  all  SCs  have  been  completed 

$29.78 

i 

520 

20 

$309,712 

0% 

24 

12480 

$926,638 

$309,712 

299% 

199% 

Table  14:  To-be  Process  Analysis 


This  scenario  represents  a  combination  of  notional  and  verified  data  to  portray  current 
activities  contained  in  the  SHIPMAIN  process  reengineered  to  maximize  utilization  of 
3D  laser  scanning  and  PLM  assets.  Not  every  sub-process  will  be  affected  in  this 
scenario;  instead,  only  affected  processes  will  be  used  for  comparison.  All  others  may  be 
assumed  static  as  described  in  their  as-is  state. 

Cost  of  3D  Terrestrial  Laser  Scanning  Technology 

The  cost  for  laser  scanning  equipment  and  required  software  was  provided  by  the  IEDP 
Project  Manager  for  SIS.  The  SISs  IEDP  Project  Manager  stated  that  the  current  cost 
has  not  changed  from  the  estimates  LT  Komoroski  used  in  her  2005  research  (B.  Tiltion, 
personal  communication,  May  16,  2007).  For  this  study,  the  cost  for  IT  used  in  LT 
Komoroski’s  2005  study  will  be  increased  by  3%  to  account  for  inflation  and  will  be 
amortized  over  a  10-year  period.  Cost  and  assumptions  for  the  3DIS  are: 

•  Current  inflation  adjusted  initial  cost  is  $90,640  for  one  3DIS  scanner  and  its 
applicable  software  suite. 

•  Maintenance/upkeep  annual  cost-estimate  is  20  percent. 
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•  Use  estimate  is  200  days  per  year. 

•  Lifespan  estimate  is  10  years. 

•  The  resulting  cost  per  unit  per  day  is:  $135.96. 

•  For  analysis  of  the  to-be  KVA  model,  this  cost  is  absorbed  by  the  actual  scanning 
process  contained  in  Block  265.1. 

The  six  planning  yards  that  support  naval  surface  force  assets  are:  Bath  Iron  Works, 
Bath,  ME;  Norfolk  Naval  Shipyard,  Norfolk,  VA;  Northrop  Grumman  Ship  Systems, 
Avondale  OP,  New  Orleans,  LA;  Northrop  Grumman  Ship  Systems,  Ingalls  OP, 
Pascagoula,  MS;  Puget  Sound  (DET)  Boston,  Boston,  MA  and;  Puget  Sound  Naval 
Shipyard,  Bremerton,  WA  (NAVSEA  Shipbuilding  Support  Office,  2007). 

To  properly  account  for  the  enterprise-wide  cost  of  the  3DIS  product,  the  daily  cost  was 
increased  by  a  factor  of  6  under  the  assumption  that  each  planning  yard  received  one 
scanner  with  the  required  software.  Accordingly,  the  daily  cost  to  introduce  3DIS  across 
the  enterprise  would  be  $815.76. 

Cost  of  PLM  Technology 

SIS  is  a  Value-added  Reseller  of  UGSs  PLM  suite  of  software  called  Teamcenter.  Under 
the  IEDP,  Teamcenter  products  will  be  introduced  to  establish  an  Integrated  Data 
Environment  using  team  collaboration  and  configuration  data-management  platforms. 
The  Teamcenter  suite  contains  the  following  specific  product  solutions:  Community 
Collaboration;  Compliance  Management;  Engineering  Process  Management;  Enterprise 
Knowledge  Management;  Lifecycle  Visualization;  Maintenance,  Repair  and  Overhaul; 
Manufacturing  Process  Management;  Portfolio  and  Program  Management;  Reporting 
and  Analytics;  Simulation  Process  Management;  Supplier  Relationship  Management, 
and  Systems  Engineering  (UGS  Corporation,  2007). 

For  the  scope  of  this  study,  Community  Collaboration,  Engineering  Process 
Management,  Lifecycle  Visualization,  Portfolio  and  Program  Management,  Reporting 
and  Analytics  and  the  Supplier  Relationship  Management  solutions  will  be  considered. 
These  solutions  will  be  part  of  the  complete  PLM  solution  evaluated  in  the  to-be  model. 
Cost  estimation  for  these  tools  has  proven  to  be  difficult.  According  to  a  leading  PLM 
provider,  “Identifying  an  accurate,  average  or  generalized  pricing  schema  for  respective 
toolsets  within  the  PLM  space  is  almost  unachievable.  It  is  safe  to  say,  however,  that 
vendor’s  price-models  have  been  decreasing  over  the  years”  (Anonymous,  personal 
communication,  June  2007). 

To  establish  a  reasonable  cost  for  the  Teamcenter  solution,  the  following  cost  estimation 
will  be  used: 

•  An  assumption  that  PLM  and  Enterprise  Resource  Planning  (ERP)  initiatives  are 
similar  in  cost  and  scope. 
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•  DoD  spent  an  average  of  $250  million  per  ERP  initiative  in  FY  06  (Service  Cost 
Estimating  Organizations,  2007). 

•  The  Department  of  the  Navy  (DoN)  budget  for  FY  06  was  $122.9  billion, 
including  supplemental  transfers  (Bozin,  2006) 

•  DoN  budget  for  Ship  Depot  Maintenance  was  $3.72  billion,  or  3  percent  of  the 
entire  DoN  budget  (Bozin,  2006). 

•  3  percent  of  a  $250  million  (the  cost  for  an  ERP)  is  $7.5  million. 

The  $7.5  million  PLM  solution  will  be  deployed  at  the  six  planning  yards  listed  earlier  in 
this  section  and  at  all  SYSCOMs/TYCOMs  supporting  surface  force  combatant  assets. 
The  cost  for  the  PLM  suite  will  be  amortized  over  10  years  with  a  2  percent  annual 
increase  for  the  cost  of  version  upgrades— bringing  the  total  cost  to  $9  million.  It  is 
assumed  that  the  PLM  software  will  be  used  230  days  per  year,  making  the  daily  cost  of 
PLM  software  $3,913.  This  cost  will  be  distributed  equally  across  all  processes  of  Phases 
IV  and  V  of  SHIPMAIN. 


To-be  Block  Assumptions  and  Data  Analysis 


Reengineering  the  to-be  scenario  proved  to  be  quite  challenging.  While  the  formal 
guidance  for  SHIPMAIN  is  relatively  mature  for  Phases  I-III,  that  is  not  so  for  Phases  IV 
and  V.  Remarkable  effort  has  been  put  into  developing  and  refining  the  business  rules 
associated  with  Phases  IV  and  V,  and  they  continue  to  be  in  a  maturing  phase  at  the 
time  of  this  study.  According  to  one  SME,  until  all  areas  become  aligned  with  the 
business  rules  and  until  the  required  technology  to  support  them  is  acquired,  the 
processes  currently  in  use  to  accomplish  the  tasks  in  Phases  IV  and  V  are  the  legacy 
procedures.  As  the  business  rules,  governance  structure  and  core  technologies  mature, 
the  processes  as  defined  in  current  SHIPMAIN  business  rules  should  become  the 
standard  practice.  In  order  to  model  the  notional  to-be  scenario,  strict  observation  of 
currently  defined  business  rules  were  coupled  with  SME  assessments  of  their  practical 
implementation  for  each  core  process.  For  additional  clarity,  all  core  processes  will  be 
described  in  terms  of  their  sub-processes  and  the  assumptions  affecting  key  parameter 
changes  from  the  as-is  to  the  to-be  scenario. 


Block  250 

Authorize  and  Issue  Letter  of  Authorization  (LOA)/Hull  Maintenance  Plan  (HMP);  Generate  2Ks 

Sub  process 

Hourly 

Personnel 

Cost 

Head 

count 

Times  Pert. 
Per  Year 

Time  to 
Complete 
(Mrs) 

Annual 

Personnel  Cost 

IT  Cost 

%IT 

ALT  (Hrs) 

Total 

Knowledge 

Total  Benefits 

Annual  Cost 

ROK 

ROI 

250.1 

Create  AHMP/EHMP 

$42.45 

0 

720 

1 

$0 

$56,250 

100% 

40 

28800 

$2,138,395 

$56,250 

3802% 

3702% 

250.2 

Create  Annual  HMP/LOA 

$42.45 

1 

1200 

40 

$2,037,678 

$56,250 

75% 

32 

153600 

$11,404,776 

$2,093,928 

545% 

445% 

250.3 

Initiate  2Ks  into  ICMP 

$35.70 

1 

624 

1 

$22,276 

$56,250 

99% 

32 

19968 

$1,482,621 

$78,526 

1888% 

1788% 

250.x 

Generate/issue  QISM 

$42.45 

2 

4 

8 

$2,717 

$56,250 

90% 

32 

2560 

$190,080 

$58,967 

322% 

222% 

Process  Totals: 

$15,215,872 

$2,287,671 

665% 

565% 

Table  15:  Block  250  (ROK) 


Assumptions  for  Block  250  are: 


•  PLM  product  suite  would  provide  the  means  for  processes  identified  in  the 
business  rules  as  “future  enhancements”  to  become  a  reality. 
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•  A  conservative  estimate  of  20  percent  greater  efficiency  was  applied  to  the  times 
fired  per  year  for  Blocks  250.1  and  205.3  due  to  automation. 


Block  265 

Hull  Installation  and  Risk  Assessment 

Sub  process 

Hourly 

Personnel 

Cost 

Head 

Times  Pert. 
Per  Year 

Complete 

(Hrs) 

Annual 

Personnel  Cost 

IT  Cost 

%IT 

ALT  (Hrs) 

Total 

Knowledge 

Total  Benefits 

Annual  Cost 

ROK 

ROI 

265.1 

Installation  Procurement,  Design  & 
Advance  Planning 

$43.10 

17 

624 

128 

$58,527,196 

$219,402 

75% 

40 

1697280 

$126,022,772 

$58,746,598 

215% 

115% 

265.2 

Hull  Installation  Readiness  Review 

$29.78 

2 

520 

32 

$991,238 

$56,250 

85% 

40 

277333 

$20,591,956 

$1,047,488 

1966% 

1866% 

265.3 

Evaluate  Maturity  Status 

$50.16 

1 

520 

20 

$521 ,696 

$56,250 

0% 

40 

20800 

$1,544,397 

$577,946 

267% 

167% 

265.4 

Provide  Risk  Assessment 

$50.16 

1 

520 

40 

$1,043,391 

$56,250 

0% 

56 

29120 

$2,162,155 

$1,099,641 

197% 

97% 

265.4.1 

Formally  Propose  Install  for 

Readniess  Assessment  and  Auth. 

$50.16 

1 

520 

20 

$626,035 

$56,250 

0% 

40 

124800 

$9,266,380 

$682,285 

1358% 

1258% 

265.5 

Risk/Readiness  Determination 

$59.01 

4 

130 

40 

$1,22/, 34/ 

$56,250 

0% 

56 

29120 

$2,162,155 

$1,283,59/ 

168% 

68% 

Process  Totals: 

$161,749,816 

$63,437,554 

255% 

155% 

Table  16:  Block  265  (ROK) 


Assumptions  for  Block  265  are: 

•  There  are  17  unique  tasks  involved  in  Block  265.1. 

•  The  15  employees  required  for  the  ship-check  task  of  Block  265.1  don’t  use  the 
entire  time  allotted  to  complete  the  process.  The  15  ship  check  employees  are 
notionally  reallocated  to  remaining  tasks  of  a  similar  pay  grade. 

•  Two  additional  employees  are  required  to  accomplish  the  17  tasks. 

•  Cycle-time  will  improve  by  a  conservative  estimate  of  20  percent  with  the 
addition  of  PLM  and  3D  laser  scanning.  PLM  will  allow  suppliers  and  purchasers 
to  share  requirements  and  plan  for  delivery  in  a  real-time,  Integrated  Data 
Environment.  3D  laser  scanning  will  provide  more  accurate  design  parameters  to 
suppliers  than  hand-drawn  images— reducing  the  amount  of  “field  engineering” 
required. 


Block  280 

Resolve  "Not  Authorized/Deferred  SC" 

Sub  process 

Hourly 

Personnel 

Cost 

Head 

count 

Times  Pert. 
Per  Year 

Time  to 
Complete 
(Hrs) 

Annual 

Personnel  Cost 

IT  Cost 

%IT 

ALT  (Hrs) 

Total 

Knowledge 

Total  Benefits 

Annual  Cost 

ROK 

ROI 

280 

Update  HMP.LOA  and  Fielding  Plan 

$29.78 

1 

520 

24 

$371,714 

$56,250 

80% 

24 

49920 

$3,706,552 

$427,964 

866% 

766% 

Block  300 

Install  SC 

Sub  process 

Hourly 

Personnel 

Cost 

Head 

count 

Times  Pert. 
Per  Year 

Time  to 
Complete 
(Hrs) 

Annual 

Personnel  Cost 

IT  Cost 

%IT 

ALT  (Hrs) 

Total 

Knowledge 

Total  Benefits 

Annual  Cost 

ROK 

ROI 

300 

Complete  installation  and  testing 

$42.45 

36 

624 

35 

$33,377,170 

$56,250 

35% 

40 

1275733 

$94,722,998 

$33,433,420 

283% 

183% 

Table  17:  Block  280  and  300  (ROK) 


Assumptions  for  Block  300  are: 

•  The  majority  of  management  and  verification  tasks  will  be  accomplished  by  30 
percent  fewer  staff  due  to  collaboration  and  access  to  the  common  data 
environment  provided  by  PLM. 
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•  Cycle-time  will  improve  by  20  percent  due  to: 

•  Improved  coordination  between  suppliers  and  the  shipyards 

•  Less  rework  due  to  installation  items  being  built  more  accurately  from  the  3D 
imagery  provided  of  as-built  configuration. 


Block  310 

Feedback:  Cost,  CM,  Performance,  Schedule,  ILS 

Sub  process 

Hourly 

Personnel 

Cost 

Head 

count 

Times  Pert. 

Per  Year 

Time  to 
Complete 
(Hrs) 

Annual 

Personnel  Cost 

IT  Cost 

%IT 

ALT  (Hrs) 

Total 

Knowledge 

Total  Benefits 

Annual  Cost 

ROK 

ROI 

310 

Provide  Feedback  Data 

$29.78 

1 

624 

10 

$185,857 

$56,250 

50% 

24 

24960 

$1,853,276 

$242,107 

765% 

665% 

Table  18:  Block  310  (ROK) 


Assumptions  for  Block  310  are: 

•  PLM  will  enable  a  50  percent  reduction  in  staff  by  having  all  related  information 
available  through  a  single  interface. 

•  Time  to  complete  the  tasks  will  be  reduced  by  75  percent  by  eliminating  lengthy  manual 
data  collection  and  aggregation. 

•  The  process  will  be  executed  20  percent  more  often  annually. 
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B.2:  CCOPS  Case  Study 

USS  Readiness  Case  Study 

The  KVA  valuation  framework  was  applied  to  the  fictitious  U.S.  Navy  warship,  USS 
Readiness.  Our  case  study  focused  on  the  cryptologic  carry-on  program  (CCOP) 
portfolio  of  intelligence  information  systems  and  in  particular,  the  ship  borne  signals 
intelligence  collection  process.  KVA+RO  allows  for  analysis  of  existing  and  future  CCOP 
systems  on  ISR  activities,  processes  and  operations  for  each  system  in  the  portfolio. 
Individual  CCOP  systems  in  the  portfolio  can  be  compared  once  baseline  data  is  created, 
enabling  decision-makers  to  make  financial  decisions  and  projections  based  on 
quantitative  data. 

Case  Study  Background 

The  USS  Readiness  is  outfitted  to  conduct  Intelligence,  Surveillance,  and 
Reconnaissance  (ISR)  missions  and  has  a  contingent  of  information  warfare  operators 
performing  intelligence  collection  processes  utilizing  CCOP  systems.  Principal  sub¬ 
processes  in  the  ICP  are  shown  in  the  following  diagram. 

The  warship  is  equipped  with  four  CCOP  systems  (A,  B,  C,  and  D).  CCOP  systems  may 
be  used  in  a  single  sub-process  or  across  sub-processes,  and  some  systems  such  as 
CCOP  A  are  highly  complex  with  multiple  subsystems.  Each  sub-process  is  further 
broken  down  into  individual  actions  that  may  be  required  to  perform  the  sub-process  in 
the  intelligence  collection  process.  For  example,  sub-process  “Target  Data  Processing” 
can  be  broken  down  into  a  number  of  human-based  tasks  requiring  no  automation. 

Applying  KVA  Methodology 

KVA  methodology  was  applied  to  quantify  the  value  added  by  CCOP  systems, 
information  warfare/cryptologic  operators,  and  the  enabling  ship  borne  system 
infrastructure  with  which  they  interact.  Value  provided  by  human  capital  elements  were 
compared  to  IT  elements  to  measure  efficiency  (productivity)  and  effectiveness 
(profitability).  All  assets,  sub-processes,  and  outputs  are  first  identified. 

•  Asset  analysis  encompasses  all  value  and  cost  data  related  to  each  asset  in  the 
process,  human  capital  or  IT  asset. 

•  Sub-process  analysis  includes  a  detailed  breakdown  of  the  ICP  to  include  the 
time-to-learn,  how  to  perform  each  sub-process,  and  number  of  executions  for 
each  sub-process. 

•  Process  outputs  are  established  via  time  to  learn  estimates,  including  the  total 
number  of  aggregated  process  outputs  and  a  surrogate  revenue  stream  used  to 
monetize  the  outputs. 

Contract  Number:  H98230-08-D-01 71  DO  02,  TO  02  RT018 

Report  No.  SERC-2010-TR-010 
09/25/2010 

UNCLASSIFIED 

139 


UNCLASSIFIED 


Asset  values  and  costs  are  then  allocated  throughout  the  sub-processes  in  which  they 
contribute  to  the  production  of  outputs.  The  time-to-learn  (knowledge  embedded  in 
each  sub-process)  is  multiplied  by  the  number  of  executions  of  that  sub-process,  and  the 
figure  serves  as  a  basis  for  revenue  allocation  at  the  sub-process  level.  Costs  are 
calculated  by  multiplying  the  time  it  takes  to  produce  the  process  output  times  the 
salary  of  those  producing  it  and  the  cost  per  usage  of  the  IT  asset.  Costing  typically  does 
not  include  the  cost  of  fixed  assets  as  these  costs  are  typically  used  as  a  constant 
weighting  factor.  Therefore,  these  costs  usually  do  not  affect  the  relative  performance 
estimates  for  the  various  sub-processes. 


P3 


P5 


Figure  23:  The  Intelligence  Collection  Process 


SUB-PROCESS  NAME 

CCOPA 

CCOP 

B 

CCOP 

c 

CCOP 

D 

PI 

Review  Request/Tasking 

X 

P2 

Determine  Op/Equip  Mix 

X 

P3 

Input  Search  Function/Coverage 
Plan 

X 

P4 

Search/Collection  Process 

X 

X 

P5 

Target  Data  Acquisition/Capture 

X 

X 

P6 

Target  Data  Processing 

X 

X 

X 

X 

P7 

Target  Data  Analysis 

X 

X 

X 

P8 

Format  Data  for  Report 

Generation 

X 

P9 

QC  Report 

X 

P10 

Transmit  Report 

X 
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Table  19:  USS  READINESS  CCOP  Systems 


Performance  ratios  such  as  ROKA  and  ROKI  can  be  calculated  after  costs  and  benefits 
for  each  sub-process  are  defined. 

Case  Study:  KVA  Results 

KVA  analysis  was  used  to  compare  two  example  sub-processes:  “Search  and  Collect” 
(P4)  and  “Format  Data  for  Report  Generation”  (P8).  Results  are  summarized  in  the 
following  tables  and  issues  were  identified  at  the  portfolio,  program  and  process  levels. 


Sub-Process 

CCOP  A 

CCOP  B 

CCOPC 

CCOP  D 

ROK 

Review  Request/Tasking 

PI 

168.54% 

168.54% 

Determine  Op/Equip  Mix 

P2 

166.86% 

166.86% 

Input  Search 

Function/Coverage  Plan 

P3 

152.91% 

152.91% 

Search/Collection  Process 

P4 

930.03% 

148.15% 

590.13% 

Target  Data 

Acquisition/Capture 

P5 

290.15% 

147.71% 

228.23% 

Target  Data  Processing 

P6 

319.39% 

162.59% 

436.13% 

28.18% 

142.41% 

Target  Data  Analysis 

P7 

149.98% 

534.76% 

34.55% 

121.42% 

Format  Data  for  Report 
Generation 

P8 

143.34% 

143.34% 

QC  Report 

P9 

315.88% 

315.88% 

Transmit  Report 

P10 

148.75% 

148.75% 

ROK  for  Total  Process 

278.59% 

152.81% 

485.44% 

31.37% 

196.27% 

Table  20:  Return  on  Knowledge  (ROK)  USS  READINESS  Summary  KVA  Results 


CCOP  D  is  a  cost-heavy  system  that  executes  very  few  times  with  negative  ROKs 
throughout  the  sample  period,  as  seen  in  Table  20. 

•  Is  CCOP  D  appropriate  for  this  platform  and  mission? 
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•  What  is  a  less  expensive  alternative  to  CCOP  D? 

•  Are  all  operators  appropriately  trained  in  the  use  of  CCOP  D? 

The  Search  and  Collect  process  (P4)  is  knowledge-intensive  requiring  IT  and  human 
capital  asset  investments  to  complete,  as  indicated  in  Table  21.  Moreover,  each  process 
output  necessitates  many  executions  of  the  sub-process. 

•  Could  an  even  higher  return  be  achieved  with  further  automated  search  and 
collection  systems  or  more  operators? 

•  Should  the  amount  of  knowledge  in  humans  and  IT  be  adjusted? 

•  Could  a  broader  range  of  training  allow  operators  to  perform  more  functions? 

The  Search  and  Collect  process  (P4)  is  a  high  performer  with  an  overall  return  of  239% 
compared  to  a  -20.37%  return  for  the  Format  Data  for  Report  Generation  process  (P  8). 

•  What  accounts  for  the  discrepancy  in  the  returns  received  on  each  process? 

The  Format  Data  for  Report  Generation  process  (P  8)  only  executes  once  per 
intelligence  report  (process  output)  with  nearly  one  third  of  all  operators  assigned  to 
this  sub-process  one  fifth  of  the  total  human  cost. 

•  What  causes  this  low  efficiency  level? 

The  Format  Data  for  Report  Generation  process  (P  8)  is  more  automated  than  P4. 

•  Could  this  process  be  further  automated  or  performed  by  other  operators  to  yield 
higher  efficiency  and  effectiveness  levels? 


Sub-Process 

CCOP  A 

CCOP  B 

CCOPC 

CCOP 

D 

ROKI 

Review  Request/Tasking 

PI 

68.54 

22.11 

Determine  Op/Equip  Mix 

P2 

66.86 

20.89 

Input  Search 
Function/Coverage  Plan 

P3 

52.91 

-18.44 

Search/Collection  Process 

P4 

830.03 

48.15 

239.01 

Target  Data 
Acquisition/Capture 

P5 

190.15 

47.71 

47.28 

Target  Data  Processing 

P6 

219.39 

62.59 

336.13 

-71.82 

36.67 
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Target  Data  Analysis 

P7 

49.98 

434.76 

-65.45 

21.25 

Fonnat  Data  for  Report 
Generation 

P8 

43.34 

-20.37 

QC  Report 

P9 

215.88 

79.19 

Transmit  Report 

P10 

48.75 

-17.37 

Metrics  for  Aggregated 

178.59 

52.81 

385.44 

68.63 

109.9 

Table  21:  Return  on  Knowledge  Investment  (ROKI)  USS  READINESS  Summary  KVA  Results 


Answers  to  these  questions  could  help  program  managers  allocate  funds  to  new  systems 
or  to  existing  systems  for  improve  products  or  to  eliminate  a  system  from  the  CCOP 
portfolio.  Results  could  also  be  used  to  tailor  manning  and  training  requirements  of  ISR 
crews  deploying  CCOP  systems.4 

Real  Options  Analysis 

Real  options  analysis  was  performed  to  determine  the  prospective  value  of  three  basic 
options  over  a  three-year  period  using  KVA  data  as  input  for  the  analysts.  Three 
potential  scenarios  were  identified. 

Results  of  the  real  options  analysis  indicate  that  Option  C  delivers  the  highest  value  at 
$15.2  million.  Although  apriori,  Options  A  and  B  were  expected  to  have  significant  cost 
savings,  it  is  possible  to  see  greater  total  value,  with  much  lower  volatility  (risk),  for 
Option  C  with  RO  analysis.  Fleet  and  Ship  Commanders  who  intuitively  preferred 
Option  C  because  it  permitted  greater  control  of  intelligence  assets  for  specific 
operations,  now  have  objective  data  to  help  them  review  their  preferred  option.  This  is 
not  to  say  that  the  other  options  might  provide  greater  strategic  value  in  the  long  run 
once  they  are  implemented  with  more  productive  CCOPs  assets  and  lower  volatility 
based  on  overcoming  the  initial  decrements  in  the  learning  curve  of  a  new  process 
implementation. 


4  This  case  study  revealed  a  few  limitations  to  implementation  of  KVA  to  the  Intelligence  Collection  Process  as  modeled  in  for  the  USS 

READINESS.  Please  see  Appendix  2  of  the  complete  case  study  for  limitations  and  issues  being  addressed. 
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Option  A 

Remote  to  Shore 

Option  B 

Direct  Support 

Option  C 

Permanent  SSES 

•  Data  viewed  from  geographically  remote  center. 

•  Intelligence  collection  processing  from 
consolidated  center  requires  less  intelligence 
personnel  on  ships. 

•  Consolidating  capabilities  into  central  center 
popular  movement  to  cut  costs  and  provide  more 
shore  based  operations  to  support  war-fighting 
capabilities. 

•  Similar  to  consolidation  of  service  operations  in 
businesses  into  larger  and  fewer  call  centers. 

•  CCOP  equipment  &  operators 
move  from  ship  to  ship 
whenever  a  ship  came  into  port 
for  maintenance,  repair  or 
modernization. 

•  Fewer  sets  of  CCOP  equipment 
and  operators  required  to 
service  intelligence  gathering 
needs  of  the  fleet. 

•  CCOP  systems  and  operators 
assigned  to  given  ships  at  all 
times. 

•  Requires  more  operators  and 
CCOP  systems. 

•  Potential  costs  increases, 
provides  more  control  of 
intelligence  capability  by  the 
ships  and  fleet  commanders. 

Table  22:  CCOP  Strategic  Scenarios 


Figure  24:  Real  Options  Analysis  of  Strategic  Scenarios 


Option  A 

Option  B 

Option  C 

PV  Option  Cost  (Year  1) 

$348,533 

$1,595,697 

$1,613,029 

PV  Option  Cost  (Year  2) 

$4,224,487 

$3,043,358 

$4,494,950 

PV  Option  Cost  (Year  3) 

$3,688,994 

$10,105,987 

$8,806,643 

PV  Revenues 

$24,416,017 

$33,909,554 

$38,820,096 
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PV  Operating  Costs 

$16,220,188 

$16,765,513 

$9,951,833 

PV  Net  Benefit 

$8,195,829 

$17,144,041 

$28,868,264 

PV  Cost  to  Purchase  Option 

$425,000 

$169,426 

$72,611 

Maturity  Years 

3.00 

3.00 

3.00 

Average  Risk-Free  Rate 

3.54% 

3.54% 

3.54% 

Dividend  Opportunity  Cost 

0.00% 

0.00% 

0.00% 

Volatility 

26.49% 

29.44% 

15.04% 

Total  Strategic  Value  with 
Options 

$1,386,355 

$4,466,540 

$15,231,813 

Table  23:  Summary  Real  Options  Analysis  Results 


Conclusions  and  Recommendations 

Applying  the  KVA+RO  framework  to  the  USS  READINESS  demonstrates  how 
defensible  and  relatively  objective  metrics  could  be  derived  for  analysis  of  each  CCOP’s 
ROI  performance  in  the  portfolio. 5  Based  on  results  of  our  initial  research,  we  make 
several  recommendations: 

•  Expand  scope  of  initial  study.  KVA  methodology  should  be  applied  and 
analyzed  over  a  larger  sampling  period  to  accurately  measure  the  impact  of  CCOP 
systems.  A  larger  study  should  be  conducted  on  CCOP  systems  at  the  Carrier  Strike 
Group  (CSG)  or  Expeditionary  Strike  Group  (ESG)  level  over  the  course  of  one 
deployment  to  begin  establishing  performance  baselines  for  systems  and  processes.  6 

•  Collect  additional  process  data.  Supplemental  data  on  human  and  automated 
processes  should  be  collected  to  attain  near  real-time  performance  data  reporting. 
Automated  logging  of  system  utilization  and  performance  are  readily  available  in 
many  business  applications.  Adapting  such  mechanisms  for  use  with  CCOP  systems 
would  facilitate  the  performance  analysis. 

•  Implement  KVA  software  for  real-time  analysis.  Although  several 
accounting  software  packages  have  included  KVA  analytical  capabilities,  the  NPS 
research  team  has  identified  GaussSoft  KVA  software  as  the  most  comprehensive 


5  KVA  analysis  was  conducted  on  a  limited  set  of  data.  To  obtain  a  more  comprehensive  picture  of  CCOP  system  contribution, 
multiple  iterations  of  this  analysis  would  have  to  be  run  across  the  Navy-wide  enterprise  of  intelligence  collection  platforms  to 
obtain  a  comprehensive  understanding  of  CCOP  program  contribution. 

6Currently  in  process  with  the  Third  Expeditionary  Strike  Group. 
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software  platform  for  conducting  the  level  of  analysis  required  by  DoD  program 
managers.  Implementing  GaussSoft  software  allows  real-time  system  and  process 
inputs  to  be  received,  as  well  as  proof-of-concept  and  a  testing  of  the  operational 
capabilities  of  the  software.  7 

•  Expand  research  study  to  include  other  public  and  private  sector 
organizations.  An  extensive  research  study  should  be  conducted  on  the  Market 
Comparables  Approach  to  include  a  valuation  study  of  the  intelligence  products 
produced  by  private  military  corporations,  along  with  competitive  and  business 
intelligence  organizations  to  achieve  a  baseline  price  per  unit  of  output  metric.  One 
of  the  study’s  primary  objectives  would  be  to  develop  universally  accepted 
descriptions  of  embedded  knowledge  and  required  learning  time  of  each  system  and 
process. 

•  An  external  organization  should  be  selected  to  maintain  KVA  databases 
for  CCOP  systems.  This  organization  would  act  as  the  central  repository  for 
system  performance  data  to  provide  reports  and  analysis  on  a  quarterly  or  semi¬ 
annual  basis  enabling  program  managers  to  make  informed  acquisition  decisions. 
This  data  could  be  expanded  to  include  other  systems  and  core  processes  to 
benchmark  performance  across  the  enterprise. 
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B.3:  Modular  Munitions  Case  Study 


Introduction 

The  DoD  is  often  criticized  for  point  design  systems  that  are  made  obsolete  by  changes 
in  the  world  situation  by  the  time  a  system  completes  development,  test  and 
deployment.  While  this  criticism  may  be  well  founded  in  some  cases,  there  are  many 
DoD  systems  that  have  been  designed  with  flexibility  in  mind,  and  they  have  been 
successfully  used  in  operations  for  many  years,  and  even  decades.  Examples  such  as 
these  can  be  used  as  case  studies  to  formalize  and  test  out  methodologies  and  tools  for 
weighing  and  valuing  flexibility  in  design  to  support  future  acquisition  decisions.  One 
promising  case  study  is  associated  with  the  air-delivered  munitions  employed  by  the 
United  States’  Air  Force  and  Navy.  Researchers  at  AFIT  have  recently  looked  at  the 
GBU-24  Laser  Guided  Bomb  (LGB)  and  GBU-31  Joint  Direct  Attack  Munition  (JDAM) 
as  part  of  their  investigations  into  modularity.  That  case  study  is  now  being  expanded  to 
support  the  present  investigations  into  flexibility  under  RT-18. 

Background 

In  the  1960’s  the  DoD  initiated  the  development  of  the  Paveway  series  of  guided  bombs. 
This  was  initiated  as  a  low  cost  initiative  to  allow  precision  delivery  of  existing 
warheads.  Initially  designed  to  work  with  the  M117  bomb,  they  were  later  adapted  for 
use  with  the  newly  designed  Mk  80  series  of  bombs.  The  Mk-82  (500  lb),  Mk-83  (1000 
lb)  and  Mk-84  (2000  lb)  bombs  are  the  warhead  component  of  many  munitions  used  by 
both  the  Air  Force  and  the  Navy  (see  tables  below).  Development  of  the  Paveway  series 
continued  in  the  1970’s  (Paveway  II)  and  1980’s  (Paveway  III)  resulting  in  a  family  of 
munitions  that  can  be  used  for  a  wide  range  of  targets  and  delivery  profiles.  The  1980s 
saw  development  of  a  new  penetrator  bomb,  the  2000  lb  BLU-109,  that  could  be  used 
with  the  Paveway  guidance  kit  to  address  sub-surface  and/or  hardened  targets.  In  1991 
the  Paveway  III  was  rapidly  modified  to  accommodate  a  hastily  developed  4700  lb 
warhead,  originally  produced  from  deactivated  howitzer  barrels  (later  produced  as  BLU- 
113,  and  later  still,  BLU-122  warheads).  This  “bunker  buster”  bomb  was  conceived, 
developed,  tested,  deployed  and  operationally  used  within  a  two  month  period!  As 
stated  above,  the  Paveway  family  of  munitions  accommodated  a  wide  range  of 
applications;  they  could  be  assembled  in  the  field  in  response  to  a  daily  Air  Tasking 
Order  using  an  array  of  options  in  terms  of  warheads,  fuzes,  guidance  kits,  and  delivery 
aircraft.  Consistent  with  definitions  provided  under  RT-18,  the  Paveway  systems  appear 
to  capture  elements  of  both  adaptability  (in  the  field)  and  flexibility  (easily  modified)  to 
achieve  new  capabilities. 

PAVEWAY  Munition  Variants  (courtesy  of  Wikipedia) 

•  GBU-10  Paveway  II  -  Mk  84  2000  lb  (907  kg)  bomb 
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•  GBU-12  Pavewav  II  -  Mk  82  500  lb  (227  kg)  bomb 

•  GBU-16  Pavewav  II  -  Mk  82  1000  lb  (454  kg)  bomb 

•  GBU-22  Pavewav  III  -  Mk  82  500  lb  (227  kg)  bomb.  Developed  at  the  same  time 
as  GBU-24,  with  some  limited  export  success,  but  was  not  adopted  by  USA  as  it 
was  felt  to  be  too  small  of  a  warhead  for  the  desired  effects  at  the  time. 

•  GBU-24  Pavewav  III  -  Mk  84/BLU-109  2000  lb  (907  kg)  class  bomb 

•  GBU-27  Pavewav  III  -  BLU-ioq  2000  lb  (907  kg)  bomb  with  penetration 
warhead,  specially  designed  for  F-117  because  the  large  fins  of  GBU-24  couldn't 
fit  into  the  bomb  bay  of  F-117. 

•  GBU-28  Pavewav  III  -The  latest  warhead  used  in  the  GBU-28/B  series  is  the 
4700  lb  BLU-122/B,  a  development  of  earlier  BLU-113  on  early  GBU-28S. 

•  Pavewav  IV  -  500  lb  (227  kg)  bomb 

In  the  1990s  the  DoD  developed  a  new  guidance  kit  using  a  combination  of  Inertial 
Navigation  (INS)  and  the  satellite  based  Global  Positioning  System  (GPS).  These  Joint 
Direct  Attack  Munition  (JDAM)  guidance  kits  (see  table  below)  were  made  to  be 
compatible  with  existing  warheads  and  fuzes,  resulting  in  an  expanded  family  of 
munitions  available  to  the  Air  Force  and  the  Navy.  As  in  the  Paveway  series  of  guided 
bomb,  they  could  be  assembled  in  the  field  according  to  the  daily  Air  Tasking  Order  and 
could  be  delivered  by  most  Air  Force  and  Navy  strike  aircraft.  The  JDAM  program  was 
considered  very  successful  by  most  accounts,  achieving  an  effective  system  at  an 
affordable  cost  with  a  relatively  short  development  time  compared  to  other  munition 
programs  being  pursued  during  the  same  period  of  time.  The  1990s  also  saw  the 
development  of  the  Advanced  Unitary  Penetrator  (BLU-116)  warhead  that  could  be  used 
with  existing  fuze  and  guidance  kits  (JDAM  and  Paveway)  to  achieve  greater  penetration 
than  that  achievable  with  the  BLU-109.  Additional  fuze  modules  were  developed  as 
well,  to  include  the  Joint  Programmable  Fuze  (FMU-152)  and  the  Hard  Target  Smart 
Fuze  (FMU-157).  In  addition  to  fielded  variants  of  guided  munitions,  numerous  proof  of 
concept  demonstrations  have  been  conducted  using  modular  components  from  the 
Paveway  and  JDAM  systems.  For  example,  prototype  demonstrations  of  “agent  defeat” 
warheads  using  incendiary  fills  have  been  conducted.  Concepts  utilizing  these  new 
warheads  would  again  make  use  of  existing  modular  components  to  reduce  risk  and 
avoid  unnecessarily  costly  development. 

JDAM  Variants  (courtesy  of  Wikipedia) 

•  2,000  lb  (900  kg)  nominal  weight 
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o  GBU-3i(V)i/B  (USAF)  Mk-84 
o  GBU-3i(V)2/B  (USN/USMC)  Mk-84 
o  GBU-3i(V)3/B  (USAF)  BLU-ioq 
o  GBU-3i(V)4/B  (USN/USMC)  BLU-ioq 

•  1,000  lb  (450  kg)  nominal  weight 

o  GBU-32(V)i/B  (USAF)  Mk-83 
o  GBU-32(V)2/B  (USN/USMC)  Mk-83 
o  GBU-35(V)i/B  (USN/USMC)  BLU-110 

•  500  lb  (225  kg)  nominal  weight 

o  GBU-38/B  (USAF)  Mk-82.(USN/USMC)Mk-82  and  BLU-111 
o  GBU-54/B  LaserJDAM  (MK-82) 

Not  all  munitions  in  the  inventory  are  modular  in  design,  so  it  may  be  useful  to  compare 
the  modular  Paveway  and  JDAM  series  to  alternative  approaches.  Possible  examples 
include  the  Small  Diameter  Bomb  (SDB)  or  the  Joint  Air-to-Surface  Standoff  Munition 
(JASSM).  Both  of  these  munitions  are  better  described  as  integral  designs.  JASSM, 
emerging  from  an  Acquisition  Reform  pilot  program  of  the  1990s,  was  developed  to 
meet  threshold  requirements  at  minimum  cost.  Consideration  of  flexibility  (or 
adaptability)  appeared  to  be  set  aside  in  favor  of  trying  to  control  cost  and  keep  the 
program  on  schedule.  Any  future  consideration  of  additional  warhead  types  for  JASSM 
(such  as  area  defeat  mechanisms)  would  likely  be  met  with  high  modification  costs 
because  of  how  the  base  warhead  was  accommodated  in  the  original  design.  In  the  case 
of  the  SDB,  very  tight  space  and  weight  constraints  to  allow  for  internal  bay  carriage 
drove  the  developers  to  a  more  integral  design.  This  is  a  well  known  tradeoff  between 
modular  and  integral  designs,  and  can  be  seen  in  consumer  products  as  well.  Examples 
include  larger  desktop  personal  computers  (more  modular  in  design  to  accommodate 
product  variety)  vs.  notebook  computers  designed  for  space  and  weight  constraints, 
yielding  more  integral  designs  that  are  not  as  easily  expanded  or  modified. 

Candidate  Case  Study  Structure 

One  of  the  questions  being  addressed  by  RT-18  is  that  of  how  to  design  a  system  (or 
justify  the  candidate  design)  so  that  its  capabilities  can  be  easily  (in  terms  of  time  and 
money)  increased  in  response  to  new  requirements  or  previously  unanticipated 
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operational  scenarios.  In  order  to  do  this,  one  needs  to  be  able  to  quantify  capability,  or 
changes  in  capability,  the  perceived  value  of  obtaining  the  new  capability,  and  the  cost  of 
achieving  the  new  capability.  The  latter  of  these,  the  realm  of  cost  estimation,  can  be 
done  using  analogous,  parametric  and  engineering  cost  approaches  for  a  given  system 
based  on  the  availability  of  data.  Models  such  as  COSYSMO  and  COPLIMO,  developed 
by  Boehm,  Laine,  et.al.,  represent  refinements  of  the  parametric  approach  and  are  being 
examined  to  accommodate  new  parameters  associated  with  flexibility.  Examples  of 
parameters  that  may  have  an  impact  on  flexibility  include,  but  certainly  are  not  limited 
to,  modularity  and  its  sub-measures  of  reconfigurability  and  extensibility.  Recent  work 
[Stryker,  Jacques,  2010]  has  demonstrated  methods  for  quantifying  these  modularity 
measures  based  on  the  functional  and  physical  architecture  of  the  system,  and 
extensions  of  these  methods  to  higher  level  concepts  are  possible. 

The  value  associated  with  a  change  in  capability  must  be  related  to  what  the  user  or 
sponsor  would  be  willing  to  pay,  or  give  up,  in  order  to  achieve  it.  This  obviously  relies 
on  quantification,  or  at  least  clarification,  of  the  changes  in  capability  associated  with 
the  development  under  consideration.  In  order  to  provide  quantification  of  capability,  a 
basis  for  comparison  must  be  constructed  that  identifies  the  different  tasks  that  can  be 
accomplished  as  well  as  the  degree  to  which  they  can  be  accomplished.  The  DoD’s 
Universal  Joint  Task  List  (UJTL),  [CJCSM3500.04C],  provides  an  authoritative  list  of 
military  tasks  as  well  as  suggested  conditions  and  measures  appropriate  for  the 
individual  tasks.  Hierarchically  organized  at  Strategic  National,  Theater,  Operational, 
and  Tactical  levels,  the  UJTL  attempts  to  provide  an  exhaustive  glossary  for  all  tasks 
that  the  military  may  need  to  perform  in  order  to  deploy,  operate  and  sustain  forces  in 
support  of  national  defense.  However,  the  very  scope  of  the  UJTL  results  in  tasks 
defined  at  a  generally  high  level  of  abstraction.  For  example,  if  we  look  for  tasks 
associated  with  the  delivery  of  munitions,  we  come  up  with  the  following  tasks  from  the 
UJTL: 

Candidate  Tasks  From  the  Universal  Joint  Task  List 
OP  3.2  Attack  Operational  Targets 

OP  3.2.1  Provide  Close  Air  Support  Integration  for  Surface  Forces 
OP  3.2.2  Conduct  Non-Lethal  Attack 
OP  3.2.4  Suppress  Enemy  Air  Defenses 
OP  3.2.5  Interdict  Operational  Forces/Targets 
OP  3. 2. 5.2  Conduct  Surface/Subsurface  Firepower  Interdiction  of  Operational 
Forces/Targets 

OP  3.2.6  Provide  Firepower  in  Support  of  Operational  Maneuver 


Note  that  these  tasks  are  very  general  and  do  not  immediately  support  quantifying 
varying  levels  of  capability  between  different  munition  systems.  A  possible  extension  of 
the  UJTL  approach  can  provide  a  more  useful  delineation  of  capability  as  follows: 
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Extended  Definitions  for  Strike  Related  Tasks 

OP  3.X.X  Defeat  fixed  surface  targets 
OP  3.X.X  Defeat  mobile  surface  targets 
OP  3.X.X  Defeat  sub-surface  targets 
OP  3.X.X  Defeat  Area  Targets 
OP  3.X.X  Defeat  Chem/Bio  Facilities 

OP  3.X.X  Limit  Collateral  Damage  (possibly  an  attribute/measure  of  other  tasks) 
OP  3.X.X  Survive  enemy  air  defenses 


Extensions  such  as  these  could  be  used  to  distinguish  capability  resulting  from  varying 
warhead  and/or  guidance  options  associated  with  a  munition  development  program 
(similar  extensions  can  be  developed  for  other  mission  areas).  Conditions,  attributes 
and  measures  associated  with  these  extended  task  definitions  also  need  to  be  defined  in 
order  to  quantitatively  compare  system  approaches.  Normalized  measures  can  be  used 
to  define  a  “weighted  capability  volume”  based  on  how  much  of  the  capability  is 
achievable  for  a  given  system  concept.  This  capability  measure  should  also  provide  the 
basis  for  the  value  of  the  capability,  although  additional  approaches  will  be  required  to 
solicit  value  assessments  from  users  or  sponsors  of  the  systems  under  consideration. 
The  Value  Driven  Design  approach  of  Collopy  and  others  will  likely  be  important  in 
establishing  the  value  associated  with  a  given  concept. 

Once  a  basis  for  comparison  of  achievable  capability  is  established,  the  design  features 
of  a  system  that  enable  it  to  achieve  those  capabilities  must  be  identified.  The  design 
features  and  other  parameters  associated  with,  for  example,  modularity  and 
interoperability,  will  be  necessary  to  support  cost  estimates.  For  the  case  of  munitions, 
recently  completed  work  can  be  extended  to  support  flexibility  assessment  under  RT-18. 
[Stryker  and  Jacques,  2010]  and  [Oyama,  Stryker,  Jacques  and  Long,  2010]  used  the 
munition  case  study  to  support  investigations  into  modularity  and  its  possible  relation 
to  rapid  assembly  and  check-out  processes.  That  work  used  functional,  physical  and 
interface  definitions  of  the  system  to  establish  modularity  measures  associated  with 
reconfigurability  and  extensibility  (and  others).  These  measures  provided  indications 
of  the  adaptability  and  ease  with  which  the  system  might  be  modified  to  achieve  new 
capabilities,  contributing  to  the  overall  flexibility  of  the  system.  While  there  are 
certainly  other  factors  affecting  flexibility  that  need  to  be  uncovered,  this  recent  work 
provides  a  useful  starting  point  for  this  investigation  under  RT-18. 

Case  Study  -  Way  Forward 

Ongoing  and  planned  extensions  to  the  existing  munition  case  study  are  as 

follows: 
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•  Using  suitable  definitions  of  tasks,  attributes  and  measures,  characterize  the 
capability  space  associated  with  the  broad  mission  area  driving  the  need  for 
strike  munitions. 

•  Identify  an  additional  munition  program  to  be  investigated  for  this  case  study. 
The  Paveway  LGB  and  the  JDAM  have  initial  work  done  on  them  in  terms  of 
functional,  physical  and  interface  definitions,  but  it  would  be  useful  to  have  a 
third  munition  program  representing  a  more  integral  design  to  compare  with  the 
modular  designs  of  Paveway  and  JDAM.  SDB  and  JASSM  were  mentioned  above 
as  candidate  systems  for  comparison,  but  AFIT  is  currently  assessing  the 
availability  of  data  to  support  the  case  study  for  those  and  other  systems. 

•  Gather  the  programmatic  data  to  support  the  case  study.  Specific  data  needs  are 
being  driven  by  the  downstream  methods  associated  with  estimating 
programmatic  flexibility,  to  include  the  COSYSMO/COPLIMO,  Hedge 
Frameworks,  or  Real  Options  valuation  models.  Anticipated  data  needs  include: 

o  Program/acquisition  structure,  to  include  planned,  and  not-previously 
planned  options  that  were  considered  and/or  exercised; 

o  Requirements  evolution,  to  include  requirements  not  accommodated  by 
design/development; 

o  Cost  evolution  for  base  programs  and  options/mods  pursued  (actual  cost 
would  be  preferable  for  the  case  study,  but  we  need  to  consider  consistent 
sourcing  of  the  data  across  all  programs  considered); 

o  Cost  estimates  for  options  not  (yet)  exercised. 

•  Quantify  the  measures  of  capability  associated  with  the  various  munitions. 

•  Gather  inputs  from  users  to  provide  an  assessment  of  value  for  the  munition 
systems  and  other  options  that  may  still  be  available  for  future  munition 
development. 

•  Evaluate  the  programmatic  flexibility  achieved  at  various  stages  of  the  life  cycle 
for  the  munitions. 
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